2024-09-21T23:15:13.568 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T23:15:13.585 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T23:15:13.861 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523 branch: squid description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/journal} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914523' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 1478 sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 sleep_before_teardown: 0 subset: 24/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjsKFqN5SC1/+BMa9J8b+HKXSYZR3yPUgK33tyW4AfS/ctwQYgeq+4ts56rDc59R96WoIpPs4k2sOxuhBEvtA4= smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5BtSD7dQLokjnkXce81rFE2SMz4ZZ0SctCFcIbe1CfQX6xWVjh/o1itejmj+PIvv4PgsqC8P18bmGK6lFugnE= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - workunit: clients: client.0: - rbd/journal.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_21:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T23:15:13.861 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-21T23:15:13.862 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-21T23:15:13.862 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T23:15:13.864 INFO:teuthology.task.internal:Checking packages... 2024-09-21T23:15:13.887 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4e45e6b490e6cbe082cdf155fea780438ec53fc7' 2024-09-21T23:15:13.887 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T23:15:13.887 INFO:teuthology.packaging:ref: None 2024-09-21T23:15:13.887 INFO:teuthology.packaging:tag: None 2024-09-21T23:15:13.887 INFO:teuthology.packaging:branch: squid 2024-09-21T23:15:13.887 INFO:teuthology.packaging:sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:15:13.887 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-21T23:15:14.133 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373.g4e45e6b4 2024-09-21T23:15:14.135 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T23:15:14.141 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T23:15:14.141 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T23:15:14.184 INFO:teuthology.task.internal:Saving configuration 2024-09-21T23:15:14.200 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T23:15:14.232 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T23:15:14.259 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 23:10:43.026992', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjsKFqN5SC1/+BMa9J8b+HKXSYZR3yPUgK33tyW4AfS/ctwQYgeq+4ts56rDc59R96WoIpPs4k2sOxuhBEvtA4='} 2024-09-21T23:15:14.284 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 23:10:43.028920', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5BtSD7dQLokjnkXce81rFE2SMz4ZZ0SctCFcIbe1CfQX6xWVjh/o1itejmj+PIvv4PgsqC8P18bmGK6lFugnE='} 2024-09-21T23:15:14.284 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T23:15:14.292 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-09-21T23:15:14.292 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-21T23:15:14.293 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T23:15:14.369 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2c9483b880>, signals=[15]) 2024-09-21T23:15:14.369 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T23:15:14.376 INFO:teuthology.task.internal:Opening connections... 2024-09-21T23:15:14.376 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:15:14.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:15:14.450 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:15:14.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:15:14.523 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T23:15:14.529 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-09-21T23:15:14.547 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-09-21T23:15:14.547 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-09-21T23:15:14.604 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-09-21T23:15:14.604 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T23:15:14.605 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T23:15:14.606 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-09-21T23:15:14.679 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-09-21T23:15:14.697 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-09-21T23:15:14.697 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-09-21T23:15:14.755 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2024-09-21T23:15:14.755 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2024-09-21T23:15:14.755 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2024-09-21T23:15:14.755 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2024-09-21T23:15:14.755 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2024-09-21T23:15:14.755 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2024-09-21T23:15:14.755 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T23:15:14.756 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2024-09-21T23:15:14.756 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2024-09-21T23:15:14.756 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T23:15:14.756 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2024-09-21T23:15:14.756 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T23:15:14.756 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T23:15:14.756 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T23:15:14.756 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-09-21T23:15:14.785 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T23:15:14.797 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T23:15:14.823 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T23:15:14.823 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-09-21T23:15:14.826 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-09-21T23:15:14.842 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T23:15:14.853 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T23:15:14.853 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-09-21T23:15:14.883 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-09-21T23:15:14.906 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T23:15:14.958 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T23:15:14.977 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T23:15:14.977 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T23:15:14.977 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T23:15:14.977 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T23:15:14.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:15:14.994 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-09-21T23:15:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:15:14.995 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-09-21T23:15:15.050 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-503.el9.x86_64 2024-09-21T23:15:15.050 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.14.0-503.el9.x86_64 2024-09-21T23:15:15.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum install -y kernel 2024-09-21T23:15:15.052 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-503.el9.x86_64 2024-09-21T23:15:15.053 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-503.el9.x86_64 2024-09-21T23:15:15.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2024-09-21T23:15:17.397 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-21T23:15:18.788 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.3 MB 00:02 2024-09-21T23:15:20.488 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 34 MB/s | 20 MB 00:00 2024-09-21T23:15:21.290 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 20 MB 00:02 2024-09-21T23:15:24.847 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.5 MB 00:00 2024-09-21T23:15:26.582 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.5 MB 00:01 2024-09-21T23:15:27.081 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-21T23:15:28.325 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 19 kB 00:00 2024-09-21T23:15:28.410 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-21T23:15:30.387 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 12 MB/s | 23 MB 00:01 2024-09-21T23:15:34.016 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T23:15:35.988 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T23:15:35.989 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T23:15:36.059 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-21T23:15:36.067 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:15:36.069 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:15:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-09-21T23:15:36.069 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:15:36.069 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:15:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T23:15:36.070 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-21T23:15:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T23:15:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T23:15:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T23:15:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:15:36.070 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:15:36.070 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:15:36.070 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2024-09-21T23:15:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:15:36.071 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 79 M 2024-09-21T23:15:36.071 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 126 M 2024-09-21T23:15:36.071 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:15:36.308 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 298 kB/s | 36 kB 00:00 2024-09-21T23:15:38.033 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T23:15:38.033 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T23:15:38.127 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:15:38.133 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:15:38.133 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-09-21T23:15:38.133 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:15:38.134 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:15:38.134 INFO:teuthology.orchestra.run.smithi060.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T23:15:38.134 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-21T23:15:38.134 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T23:15:38.134 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T23:15:38.134 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T23:15:38.134 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:15:38.134 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:15:38.134 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:15:38.134 INFO:teuthology.orchestra.run.smithi060.stdout:Install 4 Packages 2024-09-21T23:15:38.134 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:15:38.135 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 79 M 2024-09-21T23:15:38.135 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 126 M 2024-09-21T23:15:38.135 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:15:38.308 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): kernel-modules-core-5.14.0-508.el9.x86_6 14 MB/s | 29 MB 00:01 2024-09-21T23:15:38.736 INFO:teuthology.orchestra.run.smithi060.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 117 kB/s | 36 kB 00:00 2024-09-21T23:15:39.017 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 12 MB/s | 35 MB 00:02 2024-09-21T23:15:39.684 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): kernel-core-5.14.0-508.el9.x86_64.rpm 4.5 MB/s | 16 MB 00:03 2024-09-21T23:15:39.684 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:15:39.684 INFO:teuthology.orchestra.run.smithi031.stdout:Total 22 MB/s | 79 MB 00:03 2024-09-21T23:15:40.311 INFO:teuthology.orchestra.run.smithi060.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 18 MB/s | 35 MB 00:01 2024-09-21T23:15:40.418 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:15:40.712 INFO:teuthology.orchestra.run.smithi060.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 6.9 MB/s | 16 MB 00:02 2024-09-21T23:15:40.788 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:15:40.788 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:15:41.785 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:15:41.786 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:15:43.820 INFO:teuthology.orchestra.run.smithi060.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 5.6 MB/s | 29 MB 00:05 2024-09-21T23:15:43.820 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:15:43.820 INFO:teuthology.orchestra.run.smithi060.stdout:Total 14 MB/s | 79 MB 00:05 2024-09-21T23:15:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:15:44.534 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:15:44.890 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:15:44.890 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:15:45.016 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T23:15:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T23:15:45.893 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:15:45.894 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:15:47.878 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T23:15:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T23:15:48.450 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:15:49.141 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T23:15:49.289 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T23:15:52.072 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T23:15:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T23:15:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T23:15:53.182 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:15:57.336 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T23:15:57.376 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:15:58.321 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:16:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:16:37.560 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:16:37.601 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:16:37.926 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:16:37.926 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T23:16:37.926 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T23:16:37.926 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T23:16:38.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:16:38.801 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:16:38.801 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:16:38.801 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T23:16:38.801 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T23:16:38.802 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T23:16:38.802 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T23:16:38.802 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:16:38.802 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:16:39.150 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T23:16:39.242 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T23:16:39.243 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T23:16:39.243 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T23:16:39.243 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T23:16:39.243 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T23:16:39.243 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T23:16:39.243 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T23:16:39.244 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T23:16:39.302 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T23:16:39.302 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T23:16:39.302 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.06 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 82.el9 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:Group : Unspecified 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 8274670 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T23:16:39.334 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : builder@centos.org 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:16:39.335 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T23:16:39.336 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T23:16:39.336 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 5.14.0-508.el9.x86_64 2024-09-21T23:16:39.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T23:16:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2024-09-21T23:16:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T23:16:41.248 INFO:teuthology.orchestra.run.smithi031.stderr:done 2024-09-21T23:16:41.250 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-21T23:16:41.264 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.5Zjin2DzP3 2024-09-21T23:16:41.265 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5Zjin2DzP3 2024-09-21T23:16:41.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.5Zjin2DzP3 2024-09-21T23:16:41.474 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.5Zjin2DzP3 is 6KB 2024-09-21T23:16:41.493 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.5Zjin2DzP3 2024-09-21T23:16:41.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T23:16:41.569 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T23:16:41.569 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T23:16:41.569 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T23:16:41.570 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T23:16:41.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T23:16:41.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2024-09-21T23:16:41.777 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:16:41.819 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:16:42.148 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:16:42.148 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T23:16:42.148 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T23:16:42.148 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T23:16:42.970 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T23:16:42.970 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:16:42.970 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:16:42.970 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T23:16:42.970 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T23:16:42.970 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T23:16:42.971 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T23:16:42.971 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:16:42.971 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:16:43.315 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T23:16:43.411 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T23:16:43.411 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T23:16:43.411 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T23:16:43.411 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T23:16:43.411 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T23:16:43.411 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T23:16:43.412 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T23:16:43.412 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T23:16:43.470 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T23:16:43.471 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T23:16:43.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -qi grub2-tools 2024-09-21T23:16:43.501 INFO:teuthology.orchestra.run.smithi060.stdout:Name : grub2-tools 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:Epoch : 1 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:Version : 2.06 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:Release : 82.el9 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: x86_64 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:Group : Unspecified 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:Size : 8274670 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:License : GPLv3+ 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T23:16:43.502 INFO:teuthology.orchestra.run.smithi060.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout:Packager : builder@centos.org 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout:Vendor : CentOS 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout:Summary : Support tools for GRUB. 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout:Description : 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout:hardware devices. 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:16:43.503 INFO:teuthology.orchestra.run.smithi060.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T23:16:43.504 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T23:16:43.504 INFO:teuthology.task.kernel:Updating grub on smithi060 to boot 5.14.0-508.el9.x86_64 2024-09-21T23:16:43.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T23:16:44.128 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2024-09-21T23:16:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi060.stderr:done 2024-09-21T23:16:45.487 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-09-21T23:16:45.502 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.MGnoNO0VVG 2024-09-21T23:16:45.503 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MGnoNO0VVG 2024-09-21T23:16:45.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.MGnoNO0VVG 2024-09-21T23:16:45.707 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.MGnoNO0VVG is 6KB 2024-09-21T23:16:45.755 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.MGnoNO0VVG 2024-09-21T23:16:45.770 DEBUG:teuthology.orchestra.run.smithi060:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T23:16:45.831 INFO:teuthology.orchestra.run.smithi060.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T23:16:45.832 INFO:teuthology.orchestra.run.smithi060.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T23:16:45.832 INFO:teuthology.orchestra.run.smithi060.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T23:16:45.832 INFO:teuthology.orchestra.run.smithi060.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T23:16:45.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T23:16:45.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2024-09-21T23:17:11.736 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T23:17:11.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-21T23:17:11.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:17:15.985 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T23:17:15.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-21T23:17:15.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:18:11.789 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T23:18:15.991 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T23:18:20.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-21T23:18:20.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:18:21.260 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-09-21T23:18:21.759 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-21T23:18:21.759 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T23:18:21.759 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-09-21T23:18:21.778 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-508.el9.x86_64 2024-09-21T23:18:21.778 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T23:18:21.778 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T23:18:21.778 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T23:18:22.779 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T23:18:22.780 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T23:18:22.841 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2024-09-21T23:18:22.860 DEBUG:teuthology.parallel:result is None 2024-09-21T23:18:24.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-21T23:18:24.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:18:25.358 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-09-21T23:18:25.761 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-21T23:18:25.761 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T23:18:25.761 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-09-21T23:18:25.776 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-508.el9.x86_64 2024-09-21T23:18:25.776 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T23:18:25.777 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T23:18:25.777 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T23:18:26.777 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T23:18:26.777 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T23:18:26.870 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2024-09-21T23:18:26.890 DEBUG:teuthology.parallel:result is None 2024-09-21T23:18:26.890 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T23:18:26.905 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T23:18:26.905 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T23:18:26.908 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T23:18:26.924 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T23:18:26.930 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T23:18:26.965 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T23:18:26.965 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T23:18:26.968 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T23:18:26.998 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T23:18:27.004 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T23:18:27.004 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T23:18:27.036 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T23:18:27.069 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T23:18:27.072 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T23:18:27.081 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T23:18:27.083 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T23:18:27.084 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T23:18:27.091 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T23:18:27.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T23:18:27.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T23:18:27.152 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T23:18:27.159 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T23:18:27.160 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T23:18:27.195 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T23:18:27.210 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T23:18:27.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T23:18:27.352 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T23:18:27.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T23:18:27.479 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:18:27.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T23:18:27.545 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T23:18:27.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T23:18:27.637 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T23:18:27.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T23:18:27.755 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:18:27.755 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T23:18:27.817 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-09-21T23:18:27.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-09-21T23:18:27.861 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T23:18:27.905 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T23:18:28.364 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T23:18:28.372 INFO:teuthology.task.internal:Starting timer... 2024-09-21T23:18:28.372 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T23:18:28.410 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T23:18:28.442 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-21T23:18:28.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-09-21T23:18:28.484 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-09-21T23:18:28.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2024-09-21T23:18:28.563 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2024-09-21T23:18:28.564 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T23:18:28.565 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-09-21T23:18:28.612 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-09-21T23:18:28.613 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2024-09-21T23:18:28.638 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2024-09-21T23:18:28.638 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi060.front.sepia.ceph.com': 'permissive'} 2024-09-21T23:18:28.638 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T23:18:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:18:28.668 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T23:18:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:18:28.701 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T23:18:28.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-09-21T23:18:28.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2024-09-21T23:18:28.786 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T23:18:28.795 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T23:18:28.805 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T23:18:28.806 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi060.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T23:22:15.498 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi060.front.sepia.ceph.com')] 2024-09-21T23:22:15.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-21T23:22:15.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:22:15.571 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-09-21T23:22:15.655 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-21T23:22:15.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-21T23:22:15.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:22:15.728 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-09-21T23:22:15.818 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-21T23:22:15.819 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T23:22:15.834 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T23:22:15.834 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T23:22:15.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T23:22:15.837 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T23:22:15.837 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T23:22:15.867 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T23:22:15.883 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T23:22:15.890 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T23:22:15.905 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T23:22:15.909 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-09-21T23:22:15.921 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-09-21T23:22:15.932 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2024-09-21T23:22:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T23:22:15.944 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2024-09-21T23:22:15.951 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T23:22:15.959 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T23:22:15.973 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T23:22:16.009 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-09-21T23:22:16.012 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T23:22:16.012 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-09-21T23:22:16.012 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T23:22:16.012 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T23:22:16.012 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T23:22:16.012 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T23:22:16.040 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-09-21T23:22:16.043 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T23:22:16.043 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-09-21T23:22:16.043 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T23:22:16.043 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T23:22:16.043 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T23:22:16.043 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T23:22:16.044 INFO:teuthology.run_tasks:Running task install... 2024-09-21T23:22:16.052 DEBUG:teuthology.task.install:project ceph 2024-09-21T23:22:16.052 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-21T23:22:16.052 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'} 2024-09-21T23:22:16.052 INFO:teuthology.task.install:Using flavor: default 2024-09-21T23:22:16.057 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T23:22:16.057 INFO:teuthology.task.install:extra packages: [] 2024-09-21T23:22:16.057 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'tag': None, 'wait_for_package': False} 2024-09-21T23:22:16.058 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:22:16.059 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'tag': None, 'wait_for_package': False} 2024-09-21T23:22:16.059 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:22:16.203 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/ 2024-09-21T23:22:16.203 INFO:teuthology.task.install.rpm:Package version is 19.1.1-373.g4e45e6b4 2024-09-21T23:22:16.216 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/ 2024-09-21T23:22:16.216 INFO:teuthology.task.install.rpm:Package version is 19.1.1-373.g4e45e6b4 2024-09-21T23:22:16.314 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T23:22:16.315 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:22:16.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T23:22:16.324 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T23:22:16.324 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:22:16.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T23:22:16.343 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T23:22:16.343 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e45e6b490e6cbe082cdf155fea780438ec53fc7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T23:22:16.352 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T23:22:16.352 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e45e6b490e6cbe082cdf155fea780438ec53fc7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T23:22:16.409 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T23:22:16.419 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T23:22:16.487 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T23:22:16.495 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T23:22:16.553 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-09-21T23:22:16.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-09-21T23:22:16.560 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2024-09-21T23:22:16.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2024-09-21T23:22:16.798 INFO:teuthology.orchestra.run.smithi031.stdout:45 files removed 2024-09-21T23:22:16.799 INFO:teuthology.orchestra.run.smithi060.stdout:45 files removed 2024-09-21T23:22:16.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2024-09-21T23:22:16.826 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw 2024-09-21T23:22:17.389 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 314 kB/s | 84 kB 00:00 2024-09-21T23:22:17.400 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 304 kB/s | 84 kB 00:00 2024-09-21T23:22:17.681 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-21T23:22:17.693 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-21T23:22:17.957 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-21T23:22:17.968 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-21T23:22:18.427 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.3 MB 00:00 2024-09-21T23:22:18.490 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.3 MB 00:00 2024-09-21T23:22:21.695 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 20 MB 00:02 2024-09-21T23:22:21.733 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 20 MB 00:02 2024-09-21T23:22:26.287 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-21T23:22:27.417 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-21T23:22:28.038 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 19 kB 00:00 2024-09-21T23:22:28.959 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 19 kB 00:00 2024-09-21T23:22:29.517 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-21T23:22:30.471 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-21T23:22:35.089 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-21T23:22:36.111 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-21T23:22:37.360 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:22:37.380 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:22:37.380 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-21T23:22:37.380 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:22:37.380 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:22:37.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 11 M 2024-09-21T23:22:37.380 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2024-09-21T23:22:37.380 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.4 M 2024-09-21T23:22:37.380 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.2 M 2024-09-21T23:22:37.380 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-21T23:22:37.380 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.4 M 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 21 M 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 25 k 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 751 k 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 507 k 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.3 M 2024-09-21T23:22:37.381 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 45 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 136 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 162 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 321 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 300 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 99 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T23:22:37.382 INFO:teuthology.orchestra.run.smithi060.stdout:Installing weak dependencies: 2024-09-21T23:22:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T23:22:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:22:37.383 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:22:37.383 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:22:37.383 INFO:teuthology.orchestra.run.smithi060.stdout:Install 30 Packages 2024-09-21T23:22:37.383 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 2 Packages 2024-09-21T23:22:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:22:37.383 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 60 M 2024-09-21T23:22:37.383 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:22:38.074 INFO:teuthology.orchestra.run.smithi060.stdout:(1/32): ceph-base-19.1.1-373.g4e45e6b4.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-09-21T23:22:38.141 INFO:teuthology.orchestra.run.smithi060.stdout:(2/32): ceph-selinux-19.1.1-373.g4e45e6b4.el9.x 378 kB/s | 25 kB 00:00 2024-09-21T23:22:38.233 INFO:teuthology.orchestra.run.smithi060.stdout:(3/32): libcephfs2-19.1.1-373.g4e45e6b4.el9.x86 8.0 MB/s | 751 kB 00:00 2024-09-21T23:22:38.392 INFO:teuthology.orchestra.run.smithi060.stdout:(4/32): ceph-radosgw-19.1.1-373.g4e45e6b4.el9.x 13 MB/s | 11 MB 00:00 2024-09-21T23:22:38.425 INFO:teuthology.orchestra.run.smithi060.stdout:(5/32): libradosstriper1-19.1.1-373.g4e45e6b4.e 2.6 MB/s | 507 kB 00:00 2024-09-21T23:22:38.441 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:22:38.455 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:22:38.455 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-21T23:22:38.455 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:22:38.455 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 11 M 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.4 M 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.2 M 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.4 M 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 21 M 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 25 k 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 751 k 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T23:22:38.456 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 507 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.3 M 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 45 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 136 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 162 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 321 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 300 k 2024-09-21T23:22:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 99 k 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout:Install 30 Packages 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2024-09-21T23:22:38.458 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:22:38.459 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 60 M 2024-09-21T23:22:38.459 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:22:38.483 INFO:teuthology.orchestra.run.smithi060.stdout:(6/32): python3-ceph-argparse-19.1.1-373.g4e45e 775 kB/s | 45 kB 00:00 2024-09-21T23:22:38.567 INFO:teuthology.orchestra.run.smithi060.stdout:(7/32): python3-ceph-common-19.1.1-373.g4e45e6b 1.6 MB/s | 136 kB 00:00 2024-09-21T23:22:38.768 INFO:teuthology.orchestra.run.smithi060.stdout:(8/32): ceph-common-19.1.1-373.g4e45e6b4.el9.x8 18 MB/s | 21 MB 00:01 2024-09-21T23:22:38.784 INFO:teuthology.orchestra.run.smithi060.stdout:(9/32): python3-cephfs-19.1.1-373.g4e45e6b4.el9 747 kB/s | 162 kB 00:00 2024-09-21T23:22:38.868 INFO:teuthology.orchestra.run.smithi060.stdout:(10/32): librgw2-19.1.1-373.g4e45e6b4.el9.x86_6 11 MB/s | 5.3 MB 00:00 2024-09-21T23:22:38.893 INFO:teuthology.orchestra.run.smithi060.stdout:(11/32): python3-rbd-19.1.1-373.g4e45e6b4.el9.x 2.7 MB/s | 300 kB 00:00 2024-09-21T23:22:38.919 INFO:teuthology.orchestra.run.smithi060.stdout:(12/32): python3-rados-19.1.1-373.g4e45e6b4.el9 2.1 MB/s | 321 kB 00:00 2024-09-21T23:22:38.944 INFO:teuthology.orchestra.run.smithi060.stdout:(13/32): python3-rgw-19.1.1-373.g4e45e6b4.el9.x 1.3 MB/s | 99 kB 00:00 2024-09-21T23:22:39.036 INFO:teuthology.orchestra.run.smithi060.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 492 kB/s | 45 kB 00:00 2024-09-21T23:22:39.061 INFO:teuthology.orchestra.run.smithi060.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 637 kB/s | 107 kB 00:00 2024-09-21T23:22:39.086 INFO:teuthology.orchestra.run.smithi060.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 957 kB/s | 160 kB 00:00 2024-09-21T23:22:39.128 INFO:teuthology.orchestra.run.smithi060.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-21T23:22:39.154 INFO:teuthology.orchestra.run.smithi060.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-21T23:22:39.179 INFO:teuthology.orchestra.run.smithi060.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-21T23:22:39.204 INFO:teuthology.orchestra.run.smithi060.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-09-21T23:22:39.230 INFO:teuthology.orchestra.run.smithi060.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 476 kB/s | 36 kB 00:00 2024-09-21T23:22:39.245 INFO:teuthology.orchestra.run.smithi031.stdout:(1/32): ceph-base-19.1.1-373.g4e45e6b4.el9.x86_ 9.3 MB/s | 5.4 MB 00:00 2024-09-21T23:22:39.255 INFO:teuthology.orchestra.run.smithi060.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-09-21T23:22:39.312 INFO:teuthology.orchestra.run.smithi031.stdout:(2/32): ceph-selinux-19.1.1-373.g4e45e6b4.el9.x 378 kB/s | 25 kB 00:00 2024-09-21T23:22:39.322 INFO:teuthology.orchestra.run.smithi060.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-09-21T23:22:39.355 INFO:teuthology.orchestra.run.smithi060.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.4 MB/s | 308 kB 00:00 2024-09-21T23:22:39.380 INFO:teuthology.orchestra.run.smithi060.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-09-21T23:22:39.406 INFO:teuthology.orchestra.run.smithi060.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-21T23:22:39.454 INFO:teuthology.orchestra.run.smithi031.stdout:(3/32): ceph-radosgw-19.1.1-373.g4e45e6b4.el9.x 13 MB/s | 11 MB 00:00 2024-09-21T23:22:39.456 INFO:teuthology.orchestra.run.smithi060.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-21T23:22:39.495 INFO:teuthology.orchestra.run.smithi031.stdout:(4/32): libcephfs2-19.1.1-373.g4e45e6b4.el9.x86 4.0 MB/s | 751 kB 00:00 2024-09-21T23:22:39.514 INFO:teuthology.orchestra.run.smithi060.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 839 kB 00:00 2024-09-21T23:22:39.529 INFO:teuthology.orchestra.run.smithi031.stdout:(5/32): libradosstriper1-19.1.1-373.g4e45e6b4.e 6.6 MB/s | 507 kB 00:00 2024-09-21T23:22:39.588 INFO:teuthology.orchestra.run.smithi031.stdout:(6/32): python3-ceph-argparse-19.1.1-373.g4e45e 776 kB/s | 45 kB 00:00 2024-09-21T23:22:39.690 INFO:teuthology.orchestra.run.smithi060.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-09-21T23:22:39.697 INFO:teuthology.orchestra.run.smithi031.stdout:(7/32): python3-ceph-common-19.1.1-373.g4e45e6b 1.2 MB/s | 136 kB 00:00 2024-09-21T23:22:39.815 INFO:teuthology.orchestra.run.smithi060.stdout:(30/32): librados2-19.1.1-373.g4e45e6b4.el9.x86 11 MB/s | 3.4 MB 00:00 2024-09-21T23:22:39.922 INFO:teuthology.orchestra.run.smithi031.stdout:(8/32): ceph-common-19.1.1-373.g4e45e6b4.el9.x8 17 MB/s | 21 MB 00:01 2024-09-21T23:22:39.924 INFO:teuthology.orchestra.run.smithi060.stdout:(31/32): libarrow-9.0.0-11.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2024-09-21T23:22:39.947 INFO:teuthology.orchestra.run.smithi031.stdout:(9/32): python3-cephfs-19.1.1-373.g4e45e6b4.el9 648 kB/s | 162 kB 00:00 2024-09-21T23:22:39.980 INFO:teuthology.orchestra.run.smithi031.stdout:(10/32): python3-rados-19.1.1-373.g4e45e6b4.el9 5.4 MB/s | 321 kB 00:00 2024-09-21T23:22:39.990 INFO:teuthology.orchestra.run.smithi060.stdout:(32/32): librbd1-19.1.1-373.g4e45e6b4.el9.x86_6 11 MB/s | 3.2 MB 00:00 2024-09-21T23:22:39.995 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:22:39.995 INFO:teuthology.orchestra.run.smithi060.stdout:Total 23 MB/s | 60 MB 00:02 2024-09-21T23:22:40.064 INFO:teuthology.orchestra.run.smithi031.stdout:(11/32): librgw2-19.1.1-373.g4e45e6b4.el9.x86_6 9.4 MB/s | 5.3 MB 00:00 2024-09-21T23:22:40.090 INFO:teuthology.orchestra.run.smithi031.stdout:(12/32): python3-rbd-19.1.1-373.g4e45e6b4.el9.x 2.1 MB/s | 300 kB 00:00 2024-09-21T23:22:40.100 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:22:40.115 INFO:teuthology.orchestra.run.smithi031.stdout:(13/32): python3-rgw-19.1.1-373.g4e45e6b4.el9.x 743 kB/s | 99 kB 00:00 2024-09-21T23:22:40.139 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:22:40.140 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:22:40.174 INFO:teuthology.orchestra.run.smithi031.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 983 kB/s | 107 kB 00:00 2024-09-21T23:22:40.199 INFO:teuthology.orchestra.run.smithi031.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-09-21T23:22:40.215 INFO:teuthology.orchestra.run.smithi031.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-09-21T23:22:40.249 INFO:teuthology.orchestra.run.smithi031.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 42 kB 00:00 2024-09-21T23:22:40.274 INFO:teuthology.orchestra.run.smithi031.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-21T23:22:40.300 INFO:teuthology.orchestra.run.smithi031.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-09-21T23:22:40.316 INFO:teuthology.orchestra.run.smithi031.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 855 kB/s | 36 kB 00:00 2024-09-21T23:22:40.333 INFO:teuthology.orchestra.run.smithi031.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-21T23:22:40.367 INFO:teuthology.orchestra.run.smithi031.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-21T23:22:40.442 INFO:teuthology.orchestra.run.smithi031.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.4 MB/s | 308 kB 00:00 2024-09-21T23:22:40.459 INFO:teuthology.orchestra.run.smithi031.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-09-21T23:22:40.476 INFO:teuthology.orchestra.run.smithi031.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-21T23:22:40.502 INFO:teuthology.orchestra.run.smithi031.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-21T23:22:40.552 INFO:teuthology.orchestra.run.smithi031.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-21T23:22:40.594 INFO:teuthology.orchestra.run.smithi031.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-21T23:22:40.777 INFO:teuthology.orchestra.run.smithi031.stdout:(29/32): librados2-19.1.1-373.g4e45e6b4.el9.x86 19 MB/s | 3.4 MB 00:00 2024-09-21T23:22:40.819 INFO:teuthology.orchestra.run.smithi031.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-09-21T23:22:40.842 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:22:40.842 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:22:40.920 INFO:teuthology.orchestra.run.smithi031.stdout:(31/32): librbd1-19.1.1-373.g4e45e6b4.el9.x86_6 22 MB/s | 3.2 MB 00:00 2024-09-21T23:22:41.076 INFO:teuthology.orchestra.run.smithi031.stdout:(32/32): libarrow-9.0.0-11.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-09-21T23:22:41.081 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:22:41.081 INFO:teuthology.orchestra.run.smithi031.stdout:Total 23 MB/s | 60 MB 00:02 2024-09-21T23:22:41.192 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:22:41.232 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:22:41.233 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:22:41.934 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:22:41.934 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:22:42.404 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:22:42.476 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-21T23:22:42.504 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-21T23:22:42.529 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-21T23:22:42.552 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-21T23:22:42.580 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-21T23:22:42.738 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-21T23:22:42.791 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-21T23:22:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-21T23:22:43.256 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-21T23:22:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-21T23:22:43.447 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 8/34 2024-09-21T23:22:43.454 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:22:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 9/34 2024-09-21T23:22:43.535 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-21T23:22:43.537 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 9/34 2024-09-21T23:22:43.565 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-21T23:22:43.566 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-21T23:22:43.600 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-21T23:22:43.610 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 11/34 2024-09-21T23:22:43.622 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-21T23:22:43.649 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-21T23:22:43.690 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-21T23:22:43.717 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-21T23:22:43.798 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-21T23:22:43.817 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-21T23:22:43.841 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-21T23:22:43.880 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-21T23:22:43.951 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-21T23:22:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-21T23:22:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-21T23:22:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-21T23:22:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-21T23:22:44.120 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-21T23:22:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-21T23:22:44.446 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-21T23:22:44.482 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-21T23:22:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-21T23:22:44.573 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-21T23:22:44.603 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/34 2024-09-21T23:22:44.611 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-21T23:22:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-21T23:22:44.701 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 8/34 2024-09-21T23:22:44.728 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 9/34 2024-09-21T23:22:44.790 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 9/34 2024-09-21T23:22:44.819 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-21T23:22:44.863 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 11/34 2024-09-21T23:22:44.959 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-21T23:22:44.988 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-21T23:22:45.053 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-21T23:22:45.088 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-21T23:22:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-21T23:22:45.184 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-21T23:22:45.199 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-21T23:22:45.273 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-21T23:22:45.352 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/34 2024-09-21T23:22:45.355 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-21T23:22:45.387 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 28/34 2024-09-21T23:22:45.414 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-21T23:22:45.453 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-21T23:22:45.482 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-21T23:22:45.515 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-21T23:22:45.841 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-21T23:22:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-21T23:22:45.976 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-21T23:22:46.007 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/34 2024-09-21T23:22:46.052 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-21T23:22:46.559 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-21T23:22:46.579 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-21T23:22:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-21T23:22:46.715 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/34 2024-09-21T23:22:46.748 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 28/34 2024-09-21T23:22:47.820 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-21T23:22:47.858 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-21T23:22:48.470 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-21T23:22:48.504 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-21T23:22:48.594 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-21T23:22:48.594 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T23:22:48.595 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T23:22:48.595 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:22:48.629 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-21T23:22:49.533 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-21T23:22:49.577 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-21T23:22:50.195 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-21T23:22:50.239 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-21T23:22:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-21T23:22:50.336 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T23:22:50.336 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T23:22:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:22:50.374 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-21T23:22:59.344 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-21T23:22:59.345 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2024-09-21T23:22:59.345 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2024-09-21T23:22:59.345 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2024-09-21T23:22:59.345 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2024-09-21T23:22:59.345 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2024-09-21T23:22:59.345 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2024-09-21T23:22:59.345 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2024-09-21T23:22:59.345 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:00.381 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-21T23:23:00.407 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-21T23:23:00.407 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:00.407 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T23:23:00.407 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T23:23:00.408 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T23:23:00.408 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:00.409 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-21T23:23:00.483 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-21T23:23:00.483 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T23:23:01.034 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-21T23:23:01.034 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-09-21T23:23:01.034 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-09-21T23:23:01.034 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-09-21T23:23:01.034 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-09-21T23:23:01.034 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-09-21T23:23:01.034 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-09-21T23:23:01.034 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-09-21T23:23:01.034 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T23:23:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/34 2024-09-21T23:23:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-21T23:23:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-21T23:23:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-21T23:23:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/34 2024-09-21T23:23:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 8/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 9/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 10/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-21T23:23:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-21T23:23:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/34 2024-09-21T23:23:02.028 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-21T23:23:02.055 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-21T23:23:02.055 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:02.056 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T23:23:02.056 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T23:23:02.056 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T23:23:02.056 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:02.058 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-21T23:23:02.155 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-21T23:23:02.156 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T23:23:02.828 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T23:23:02.828 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:02.828 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2024-09-21T23:23:02.828 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.828 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.828 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:23:02.828 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T23:23:02.828 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.828 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.828 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T23:23:02.829 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T23:23:02.830 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:02.831 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:23:03.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-test 2024-09-21T23:23:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 8/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 9/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 10/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-21T23:23:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-21T23:23:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-21T23:23:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-21T23:23:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-21T23:23:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-21T23:23:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-21T23:23:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/34 2024-09-21T23:23:03.739 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:28 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:23:03.884 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:23:03.885 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:03.885 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-21T23:23:03.885 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:03.885 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:23:03.885 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 49 M 2024-09-21T23:23:03.886 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-21T23:23:03.886 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 163 k 2024-09-21T23:23:03.886 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T23:23:03.886 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T23:23:03.886 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:03.886 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:23:03.886 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:03.886 INFO:teuthology.orchestra.run.smithi060.stdout:Install 4 Packages 2024-09-21T23:23:03.886 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:03.887 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 49 M 2024-09-21T23:23:03.887 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 209 M 2024-09-21T23:23:03.887 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:23:04.171 INFO:teuthology.orchestra.run.smithi060.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.8 MB/s | 304 kB 00:00 2024-09-21T23:23:04.204 INFO:teuthology.orchestra.run.smithi060.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.206 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T23:23:04.207 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:04.208 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:23:04.254 INFO:teuthology.orchestra.run.smithi060.stdout:(3/4): libcephsqlite-19.1.1-373.g4e45e6b4.el9.x 667 kB/s | 163 kB 00:00 2024-09-21T23:23:04.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2024-09-21T23:23:05.136 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:29 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:23:05.273 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:23:05.274 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-21T23:23:05.274 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:05.274 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:23:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 49 M 2024-09-21T23:23:05.274 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-21T23:23:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 163 k 2024-09-21T23:23:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T23:23:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T23:23:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:05.274 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:23:05.275 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:05.275 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2024-09-21T23:23:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:05.275 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 49 M 2024-09-21T23:23:05.276 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 209 M 2024-09-21T23:23:05.276 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:23:05.520 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-21T23:23:05.546 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): libcephsqlite-19.1.1-373.g4e45e6b4.el9.x 745 kB/s | 163 kB 00:00 2024-09-21T23:23:05.546 INFO:teuthology.orchestra.run.smithi060.stdout:(4/4): ceph-test-19.1.1-373.g4e45e6b4.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-21T23:23:05.548 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:05.549 INFO:teuthology.orchestra.run.smithi060.stdout:Total 30 MB/s | 49 MB 00:01 2024-09-21T23:23:05.571 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T23:23:05.577 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:23:05.599 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:23:05.599 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:23:05.989 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:23:05.989 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:23:06.539 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:23:06.614 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T23:23:06.654 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T23:23:06.681 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/4 2024-09-21T23:23:06.846 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): ceph-test-19.1.1-373.g4e45e6b4.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-21T23:23:06.848 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:06.849 INFO:teuthology.orchestra.run.smithi031.stdout:Total 31 MB/s | 49 MB 00:01 2024-09-21T23:23:06.877 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:23:06.900 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:23:06.900 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:23:07.292 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:23:07.293 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:23:07.980 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:23:08.062 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T23:23:08.104 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T23:23:08.131 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/4 2024-09-21T23:23:11.555 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/4 2024-09-21T23:23:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/4 2024-09-21T23:23:13.071 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/4 2024-09-21T23:23:13.157 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/4 2024-09-21T23:23:14.099 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/4 2024-09-21T23:23:14.099 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/4 2024-09-21T23:23:14.099 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/4 2024-09-21T23:23:14.099 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T23:23:14.414 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T23:23:14.414 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:14.414 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:23:14.415 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:14.415 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:14.415 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T23:23:14.415 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T23:23:14.415 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:14.415 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:23:14.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph 2024-09-21T23:23:15.290 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:40 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:23:15.430 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:23:15.433 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-21T23:23:15.433 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:15.433 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:23:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 6.6 k 2024-09-21T23:23:15.433 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-21T23:23:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 2.4 M 2024-09-21T23:23:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 1.5 M 2024-09-21T23:23:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 249 k 2024-09-21T23:23:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 4.7 M 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 17 M 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T23:23:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T23:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T23:23:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T23:23:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T23:23:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T23:23:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T23:23:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T23:23:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T23:23:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:15.436 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:23:15.436 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:15.436 INFO:teuthology.orchestra.run.smithi060.stdout:Install 37 Packages 2024-09-21T23:23:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:15.437 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 30 M 2024-09-21T23:23:15.437 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 108 M 2024-09-21T23:23:15.437 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:23:15.487 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/4 2024-09-21T23:23:15.487 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/4 2024-09-21T23:23:15.487 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/4 2024-09-21T23:23:15.487 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T23:23:15.870 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T23:23:15.870 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:15.870 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:23:15.870 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:15.870 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:15.870 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T23:23:15.870 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T23:23:15.870 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:15.870 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:23:15.939 INFO:teuthology.orchestra.run.smithi060.stdout:(1/37): ceph-19.1.1-373.g4e45e6b4.el9.x86_64.rp 57 kB/s | 6.6 kB 00:00 2024-09-21T23:23:16.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2024-09-21T23:23:16.156 INFO:teuthology.orchestra.run.smithi060.stdout:(2/37): ceph-mds-19.1.1-373.g4e45e6b4.el9.x86_6 7.2 MB/s | 2.4 MB 00:00 2024-09-21T23:23:16.198 INFO:teuthology.orchestra.run.smithi060.stdout:(3/37): ceph-mgr-19.1.1-373.g4e45e6b4.el9.x86_6 4.1 MB/s | 1.5 MB 00:00 2024-09-21T23:23:16.274 INFO:teuthology.orchestra.run.smithi060.stdout:(4/37): ceph-mgr-modules-core-19.1.1-373.g4e45e 3.3 MB/s | 249 kB 00:00 2024-09-21T23:23:16.365 INFO:teuthology.orchestra.run.smithi060.stdout:(5/37): ceph-mon-19.1.1-373.g4e45e6b4.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-21T23:23:16.524 INFO:teuthology.orchestra.run.smithi060.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-21T23:23:16.574 INFO:teuthology.orchestra.run.smithi060.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-21T23:23:16.750 INFO:teuthology.orchestra.run.smithi060.stdout:(8/37): ceph-osd-19.1.1-373.g4e45e6b4.el9.x86_6 28 MB/s | 17 MB 00:00 2024-09-21T23:23:16.755 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:40 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:23:16.775 INFO:teuthology.orchestra.run.smithi060.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 675 kB/s | 135 kB 00:00 2024-09-21T23:23:16.817 INFO:teuthology.orchestra.run.smithi060.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.7 MB/s | 1.2 MB 00:00 2024-09-21T23:23:16.867 INFO:teuthology.orchestra.run.smithi060.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-09-21T23:23:16.904 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:23:16.906 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:16.906 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 6.6 k 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 2.4 M 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 1.5 M 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 249 k 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 4.7 M 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 17 M 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T23:23:16.907 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T23:23:16.908 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:23:16.909 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:16.910 INFO:teuthology.orchestra.run.smithi031.stdout:Install 37 Packages 2024-09-21T23:23:16.910 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:16.910 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 30 M 2024-09-21T23:23:16.910 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 108 M 2024-09-21T23:23:16.910 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:23:16.959 INFO:teuthology.orchestra.run.smithi060.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-09-21T23:23:16.984 INFO:teuthology.orchestra.run.smithi060.stdout:(13/37): python3-requests-2.25.1-8.el9.noarch.r 533 kB/s | 125 kB 00:00 2024-09-21T23:23:17.009 INFO:teuthology.orchestra.run.smithi060.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 693 kB/s | 35 kB 00:00 2024-09-21T23:23:17.035 INFO:teuthology.orchestra.run.smithi060.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-21T23:23:17.060 INFO:teuthology.orchestra.run.smithi060.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-21T23:23:17.086 INFO:teuthology.orchestra.run.smithi060.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 826 kB/s | 42 kB 00:00 2024-09-21T23:23:17.103 INFO:teuthology.orchestra.run.smithi060.stdout:(18/37): python3-packaging-20.9-5.el9.noarch.rp 654 kB/s | 77 kB 00:00 2024-09-21T23:23:17.161 INFO:teuthology.orchestra.run.smithi060.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-21T23:23:17.186 INFO:teuthology.orchestra.run.smithi060.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-21T23:23:17.212 INFO:teuthology.orchestra.run.smithi060.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 399 kB/s | 43 kB 00:00 2024-09-21T23:23:17.245 INFO:teuthology.orchestra.run.smithi060.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-21T23:23:17.262 INFO:teuthology.orchestra.run.smithi060.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 254 kB/s | 11 kB 00:00 2024-09-21T23:23:17.287 INFO:teuthology.orchestra.run.smithi060.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.0 MB/s | 23 kB 00:00 2024-09-21T23:23:17.313 INFO:teuthology.orchestra.run.smithi060.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 783 kB/s | 19 kB 00:00 2024-09-21T23:23:17.346 INFO:teuthology.orchestra.run.smithi060.stdout:(26/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-09-21T23:23:17.371 INFO:teuthology.orchestra.run.smithi060.stdout:(27/37): python3-jaraco-classes-3.2.1-5.el9.noa 165 kB/s | 18 kB 00:00 2024-09-21T23:23:17.388 INFO:teuthology.orchestra.run.smithi060.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 264 kB/s | 20 kB 00:00 2024-09-21T23:23:17.425 INFO:teuthology.orchestra.run.smithi060.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T23:23:17.451 INFO:teuthology.orchestra.run.smithi060.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 657 kB/s | 16 kB 00:00 2024-09-21T23:23:17.476 INFO:teuthology.orchestra.run.smithi060.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-21T23:23:17.501 INFO:teuthology.orchestra.run.smithi060.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-09-21T23:23:17.526 INFO:teuthology.orchestra.run.smithi060.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-09-21T23:23:17.547 INFO:teuthology.orchestra.run.smithi031.stdout:(1/37): ceph-19.1.1-373.g4e45e6b4.el9.x86_64.rp 50 kB/s | 6.6 kB 00:00 2024-09-21T23:23:17.552 INFO:teuthology.orchestra.run.smithi060.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-21T23:23:17.577 INFO:teuthology.orchestra.run.smithi060.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 806 kB/s | 20 kB 00:00 2024-09-21T23:23:17.619 INFO:teuthology.orchestra.run.smithi060.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-21T23:23:17.661 INFO:teuthology.orchestra.run.smithi060.stdout:(37/37): python3-logutils-0.3.5-21.el9.noarch.r 151 kB/s | 46 kB 00:00 2024-09-21T23:23:17.664 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:17.664 INFO:teuthology.orchestra.run.smithi060.stdout:Total 14 MB/s | 30 MB 00:02 2024-09-21T23:23:17.755 INFO:teuthology.orchestra.run.smithi031.stdout:(2/37): ceph-mgr-19.1.1-373.g4e45e6b4.el9.x86_6 4.5 MB/s | 1.5 MB 00:00 2024-09-21T23:23:17.814 INFO:teuthology.orchestra.run.smithi031.stdout:(3/37): ceph-mds-19.1.1-373.g4e45e6b4.el9.x86_6 6.0 MB/s | 2.4 MB 00:00 2024-09-21T23:23:17.823 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:23:17.868 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:23:17.868 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:23:17.906 INFO:teuthology.orchestra.run.smithi031.stdout:(4/37): ceph-mon-19.1.1-373.g4e45e6b4.el9.x86_6 13 MB/s | 4.7 MB 00:00 2024-09-21T23:23:17.940 INFO:teuthology.orchestra.run.smithi031.stdout:(5/37): ceph-mgr-modules-core-19.1.1-373.g4e45e 1.9 MB/s | 249 kB 00:00 2024-09-21T23:23:18.098 INFO:teuthology.orchestra.run.smithi031.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-21T23:23:18.148 INFO:teuthology.orchestra.run.smithi031.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-21T23:23:18.198 INFO:teuthology.orchestra.run.smithi031.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-21T23:23:18.232 INFO:teuthology.orchestra.run.smithi031.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 4.2 MB/s | 1.2 MB 00:00 2024-09-21T23:23:18.257 INFO:teuthology.orchestra.run.smithi031.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-21T23:23:18.299 INFO:teuthology.orchestra.run.smithi031.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-21T23:23:18.382 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:23:18.383 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:23:18.475 INFO:teuthology.orchestra.run.smithi031.stdout:(12/37): ceph-osd-19.1.1-373.g4e45e6b4.el9.x86_ 23 MB/s | 17 MB 00:00 2024-09-21T23:23:18.583 INFO:teuthology.orchestra.run.smithi031.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 579 kB/s | 188 kB 00:00 2024-09-21T23:23:18.608 INFO:teuthology.orchestra.run.smithi031.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 261 kB/s | 35 kB 00:00 2024-09-21T23:23:18.634 INFO:teuthology.orchestra.run.smithi031.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-21T23:23:18.667 INFO:teuthology.orchestra.run.smithi031.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 468 kB/s | 172 kB 00:00 2024-09-21T23:23:18.693 INFO:teuthology.orchestra.run.smithi031.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 607 kB/s | 51 kB 00:00 2024-09-21T23:23:18.713 INFO:teuthology.orchestra.run.smithi031.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 528 kB/s | 42 kB 00:00 2024-09-21T23:23:18.780 INFO:teuthology.orchestra.run.smithi031.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 652 kB/s | 43 kB 00:00 2024-09-21T23:23:18.805 INFO:teuthology.orchestra.run.smithi031.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-09-21T23:23:18.847 INFO:teuthology.orchestra.run.smithi031.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-21T23:23:18.864 INFO:teuthology.orchestra.run.smithi031.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 649 kB/s | 11 kB 00:00 2024-09-21T23:23:18.881 INFO:teuthology.orchestra.run.smithi031.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-21T23:23:18.906 INFO:teuthology.orchestra.run.smithi031.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 938 kB/s | 23 kB 00:00 2024-09-21T23:23:18.931 INFO:teuthology.orchestra.run.smithi031.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-21T23:23:18.957 INFO:teuthology.orchestra.run.smithi031.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-21T23:23:18.974 INFO:teuthology.orchestra.run.smithi031.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 582 kB/s | 20 kB 00:00 2024-09-21T23:23:18.990 INFO:teuthology.orchestra.run.smithi031.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-21T23:23:19.024 INFO:teuthology.orchestra.run.smithi031.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T23:23:19.049 INFO:teuthology.orchestra.run.smithi031.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 684 kB/s | 16 kB 00:00 2024-09-21T23:23:19.083 INFO:teuthology.orchestra.run.smithi031.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2024-09-21T23:23:19.108 INFO:teuthology.orchestra.run.smithi031.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-09-21T23:23:19.133 INFO:teuthology.orchestra.run.smithi031.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-09-21T23:23:19.159 INFO:teuthology.orchestra.run.smithi031.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-21T23:23:19.176 INFO:teuthology.orchestra.run.smithi031.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-21T23:23:19.226 INFO:teuthology.orchestra.run.smithi031.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-21T23:23:19.268 INFO:teuthology.orchestra.run.smithi031.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 37 kB/s | 22 kB 00:00 2024-09-21T23:23:19.272 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:19.272 INFO:teuthology.orchestra.run.smithi031.stdout:Total 13 MB/s | 30 MB 00:02 2024-09-21T23:23:19.431 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:23:19.476 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:23:19.476 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:23:19.920 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:23:19.964 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:23:19.964 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:23:19.981 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T23:23:20.015 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T23:23:20.041 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T23:23:20.117 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T23:23:20.166 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T23:23:20.199 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T23:23:20.302 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T23:23:20.381 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T23:23:20.427 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T23:23:20.461 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T23:23:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T23:23:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T23:23:20.559 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T23:23:20.600 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T23:23:20.643 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T23:23:20.677 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T23:23:20.726 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T23:23:20.766 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T23:23:20.843 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T23:23:21.112 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T23:23:21.202 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T23:23:21.378 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T23:23:21.425 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T23:23:21.477 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:23:21.483 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T23:23:21.546 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T23:23:21.547 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T23:23:21.579 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T23:23:21.607 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T23:23:21.682 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T23:23:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T23:23:21.745 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T23:23:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T23:23:21.795 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T23:23:21.869 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T23:23:21.876 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T23:23:21.947 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T23:23:21.965 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T23:23:22.117 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T23:23:22.119 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T23:23:22.152 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T23:23:22.189 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T23:23:22.232 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T23:23:22.243 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T23:23:22.265 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T23:23:22.307 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T23:23:22.358 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T23:23:22.400 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T23:23:22.415 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 32/37 2024-09-21T23:23:22.445 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/37 2024-09-21T23:23:22.450 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T23:23:22.473 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/37 2024-09-21T23:23:22.473 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:22.473 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T23:23:22.473 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T23:23:22.473 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T23:23:22.473 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:22.489 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T23:23:22.566 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T23:23:22.721 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T23:23:22.811 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T23:23:22.869 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T23:23:22.907 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T23:23:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T23:23:23.037 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T23:23:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T23:23:23.307 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T23:23:23.384 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T23:23:23.482 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T23:23:23.526 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T23:23:23.650 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T23:23:23.814 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 32/37 2024-09-21T23:23:23.844 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/37 2024-09-21T23:23:23.872 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/37 2024-09-21T23:23:23.872 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:23.872 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T23:23:23.872 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T23:23:23.872 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T23:23:23.872 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:23.997 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 34/37 2024-09-21T23:23:24.026 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 34/37 2024-09-21T23:23:24.026 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:24.026 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T23:23:24.026 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T23:23:24.026 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T23:23:24.027 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:24.489 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/37 2024-09-21T23:23:24.524 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/37 2024-09-21T23:23:24.524 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:24.524 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T23:23:24.524 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T23:23:24.524 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T23:23:24.524 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:24.765 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/37 2024-09-21T23:23:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/37 2024-09-21T23:23:24.790 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:24.790 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T23:23:24.791 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T23:23:24.791 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T23:23:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:24.856 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/37 2024-09-21T23:23:25.412 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 34/37 2024-09-21T23:23:25.441 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 34/37 2024-09-21T23:23:25.441 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:25.441 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T23:23:25.441 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T23:23:25.441 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T23:23:25.441 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:25.904 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/37 2024-09-21T23:23:25.930 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/37 2024-09-21T23:23:25.930 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:25.930 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T23:23:25.930 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T23:23:25.931 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T23:23:25.931 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:26.163 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/37 2024-09-21T23:23:26.188 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/37 2024-09-21T23:23:26.188 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:26.188 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T23:23:26.189 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T23:23:26.189 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T23:23:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:26.255 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/37 2024-09-21T23:23:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/37 2024-09-21T23:23:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/37 2024-09-21T23:23:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/37 2024-09-21T23:23:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/37 2024-09-21T23:23:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/37 2024-09-21T23:23:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/37 2024-09-21T23:23:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 6/37 2024-09-21T23:23:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T23:23:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T23:23:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T23:23:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T23:23:26.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T23:23:26.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T23:23:26.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T23:23:26.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T23:23:26.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T23:23:26.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T23:23:26.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T23:23:26.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T23:23:26.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T23:23:26.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T23:23:26.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T23:23:26.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T23:23:26.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T23:23:27.553 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T23:23:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T23:23:27.555 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:27.556 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:23:27.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-base 2024-09-21T23:23:28.322 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/37 2024-09-21T23:23:28.322 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/37 2024-09-21T23:23:28.322 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 6/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T23:23:28.323 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T23:23:28.324 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T23:23:28.325 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T23:23:28.325 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T23:23:28.325 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T23:23:28.325 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T23:23:28.325 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T23:23:28.325 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T23:23:28.570 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:23:28.662 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:23:28.706 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:23:28.707 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-21T23:23:28.707 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:23:28.776 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install cephadm 2024-09-21T23:23:29.300 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T23:23:29.300 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T23:23:29.301 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T23:23:29.302 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:29.303 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:23:29.381 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:23:29.519 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:23:29.520 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:29.520 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-21T23:23:29.520 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:29.520 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:23:29.520 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 765 k 2024-09-21T23:23:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:29.521 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:23:29.521 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:29.521 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-21T23:23:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:29.521 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 765 k 2024-09-21T23:23:29.521 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 772 k 2024-09-21T23:23:29.521 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:23:29.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2024-09-21T23:23:29.782 INFO:teuthology.orchestra.run.smithi060.stdout:cephadm-19.1.1-373.g4e45e6b4.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-09-21T23:23:29.782 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:29.782 INFO:teuthology.orchestra.run.smithi060.stdout:Total 2.9 MB/s | 765 kB 00:00 2024-09-21T23:23:29.783 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:23:29.788 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:23:29.788 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:23:29.794 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:23:29.795 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:23:29.868 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:23:30.282 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:23:30.344 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:23:30.375 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:23:30.419 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:23:30.421 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-21T23:23:30.421 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:23:30.447 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:23:30.492 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2024-09-21T23:23:30.807 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:23:31.057 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:23:31.057 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:31.057 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:23:31.057 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:31.057 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:31.057 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:23:31.098 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:23:31.235 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:23:31.236 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:31.236 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-21T23:23:31.236 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:31.236 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:23:31.236 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 765 k 2024-09-21T23:23:31.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:31.236 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:23:31.236 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:31.236 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-21T23:23:31.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:31.237 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 765 k 2024-09-21T23:23:31.237 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 772 k 2024-09-21T23:23:31.237 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:23:31.262 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T23:23:31.503 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-19.1.1-373.g4e45e6b4.el9.noarch.rpm 2.8 MB/s | 765 kB 00:00 2024-09-21T23:23:31.504 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:31.504 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.8 MB/s | 765 kB 00:00 2024-09-21T23:23:31.504 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:23:31.510 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:23:31.510 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:23:31.516 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:23:31.516 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:23:31.593 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:23:31.874 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:23:31.984 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:23:32.013 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:23:32.014 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:32.014 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-09-21T23:23:32.014 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:32.014 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:23:32.014 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 145 k 2024-09-21T23:23:32.014 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:32.014 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:23:32.014 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:32.014 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-21T23:23:32.015 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:32.015 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 145 k 2024-09-21T23:23:32.015 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 435 k 2024-09-21T23:23:32.015 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:23:32.076 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:23:32.229 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-immutable-object-cache-19.1.1-373.g4e45e6b 678 kB/s | 145 kB 00:00 2024-09-21T23:23:32.230 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:32.230 INFO:teuthology.orchestra.run.smithi060.stdout:Total 673 kB/s | 145 kB 00:00 2024-09-21T23:23:32.230 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:23:32.238 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:23:32.238 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:23:32.299 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:23:32.299 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:23:32.447 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:23:32.450 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:23:32.522 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-21T23:23:32.549 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-21T23:23:32.549 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:32.549 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T23:23:32.550 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:32.796 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:23:32.796 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:32.796 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:23:32.796 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:32.797 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:32.797 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:23:33.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T23:23:33.452 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-21T23:23:33.452 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:33.452 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:23:33.452 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:33.452 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:33.452 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:23:33.604 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:23:33.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr 2024-09-21T23:23:33.746 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:23:33.747 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:33.747 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-09-21T23:23:33.747 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:33.747 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:23:33.747 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 145 k 2024-09-21T23:23:33.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:33.747 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:23:33.747 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:33.747 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-21T23:23:33.748 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:33.748 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 145 k 2024-09-21T23:23:33.748 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 435 k 2024-09-21T23:23:33.748 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:23:33.966 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-19.1.1-373.g4e45e6b 665 kB/s | 145 kB 00:00 2024-09-21T23:23:33.967 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:33.967 INFO:teuthology.orchestra.run.smithi031.stdout:Total 660 kB/s | 145 kB 00:00 2024-09-21T23:23:33.967 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:23:33.975 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:23:33.976 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:23:34.036 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:23:34.036 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:23:34.182 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:23:34.303 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:23:34.375 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-21T23:23:34.396 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:23:34.402 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-21T23:23:34.402 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:23:34.402 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T23:23:34.402 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:34.441 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:23:34.442 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-21T23:23:34.443 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:23:34.506 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T23:23:35.110 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:00 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:23:35.248 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:23:35.249 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:35.249 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-21T23:23:35.249 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:35.249 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:23:35.249 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 3.6 M 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 32 k 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 16 k 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T23:23:35.250 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T23:23:35.251 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:35.251 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:23:35.251 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:35.251 INFO:teuthology.orchestra.run.smithi060.stdout:Install 12 Packages 2024-09-21T23:23:35.251 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:35.251 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 8.7 M 2024-09-21T23:23:35.251 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 105 M 2024-09-21T23:23:35.251 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:23:35.392 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-21T23:23:35.392 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:35.392 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:23:35.392 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:23:35.392 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:35.392 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:23:35.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2024-09-21T23:23:35.983 INFO:teuthology.orchestra.run.smithi060.stdout:(1/12): ceph-grafana-dashboards-19.1.1-373.g4e4 210 kB/s | 32 kB 00:00 2024-09-21T23:23:36.008 INFO:teuthology.orchestra.run.smithi060.stdout:(2/12): ceph-prometheus-alerts-19.1.1-373.g4e45 93 kB/s | 16 kB 00:00 2024-09-21T23:23:36.175 INFO:teuthology.orchestra.run.smithi060.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 269 kB 00:00 2024-09-21T23:23:36.214 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:00 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:23:36.242 INFO:teuthology.orchestra.run.smithi060.stdout:(4/12): ceph-mgr-dashboard-19.1.1-373.g4e45e6b4 8.7 MB/s | 3.6 MB 00:00 2024-09-21T23:23:36.307 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:23:36.309 INFO:teuthology.orchestra.run.smithi060.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-09-21T23:23:36.352 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:23:36.353 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-21T23:23:36.353 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:23:36.359 INFO:teuthology.orchestra.run.smithi060.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-09-21T23:23:36.393 INFO:teuthology.orchestra.run.smithi060.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.6 MB/s | 548 kB 00:00 2024-09-21T23:23:36.443 INFO:teuthology.orchestra.run.smithi060.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.1 MB/s | 863 kB 00:00 2024-09-21T23:23:36.455 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T23:23:36.468 INFO:teuthology.orchestra.run.smithi060.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.9 MB/s | 144 kB 00:00 2024-09-21T23:23:36.519 INFO:teuthology.orchestra.run.smithi060.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 409 kB/s | 31 kB 00:00 2024-09-21T23:23:36.544 INFO:teuthology.orchestra.run.smithi060.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.7 MB/s | 188 kB 00:00 2024-09-21T23:23:36.677 INFO:teuthology.orchestra.run.smithi060.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.4 MB/s | 2.0 MB 00:00 2024-09-21T23:23:36.682 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:36.682 INFO:teuthology.orchestra.run.smithi060.stdout:Total 6.1 MB/s | 8.7 MB 00:01 2024-09-21T23:23:36.737 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:23:36.757 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:23:36.757 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:23:36.968 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:23:36.968 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:23:37.062 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:01 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:23:37.200 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:23:37.201 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:37.201 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-21T23:23:37.201 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:37.201 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:23:37.201 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 3.6 M 2024-09-21T23:23:37.201 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-21T23:23:37.201 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T23:23:37.201 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 32 k 2024-09-21T23:23:37.201 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 16 k 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout:Install 12 Packages 2024-09-21T23:23:37.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:37.203 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 8.7 M 2024-09-21T23:23:37.203 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 105 M 2024-09-21T23:23:37.203 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:23:37.505 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:23:37.579 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T23:23:37.622 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T23:23:37.661 INFO:teuthology.orchestra.run.smithi031.stdout:(1/12): ceph-grafana-dashboards-19.1.1-373.g4e4 215 kB/s | 32 kB 00:00 2024-09-21T23:23:37.670 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T23:23:37.686 INFO:teuthology.orchestra.run.smithi031.stdout:(2/12): ceph-prometheus-alerts-19.1.1-373.g4e45 95 kB/s | 16 kB 00:00 2024-09-21T23:23:37.707 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T23:23:37.803 INFO:teuthology.orchestra.run.smithi031.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.3 MB/s | 269 kB 00:00 2024-09-21T23:23:37.826 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T23:23:37.853 INFO:teuthology.orchestra.run.smithi031.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-09-21T23:23:37.937 INFO:teuthology.orchestra.run.smithi031.stdout:(5/12): ceph-mgr-dashboard-19.1.1-373.g4e45e6b4 8.4 MB/s | 3.6 MB 00:00 2024-09-21T23:23:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T23:23:37.987 INFO:teuthology.orchestra.run.smithi031.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 391 kB/s | 19 kB 00:00 2024-09-21T23:23:38.020 INFO:teuthology.orchestra.run.smithi031.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.2 MB/s | 548 kB 00:00 2024-09-21T23:23:38.063 INFO:teuthology.orchestra.run.smithi031.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-09-21T23:23:38.065 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T23:23:38.096 INFO:teuthology.orchestra.run.smithi031.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 923 kB/s | 31 kB 00:00 2024-09-21T23:23:38.113 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T23:23:38.130 INFO:teuthology.orchestra.run.smithi031.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.6 MB/s | 863 kB 00:00 2024-09-21T23:23:38.140 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T23:23:38.163 INFO:teuthology.orchestra.run.smithi031.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.7 MB/s | 188 kB 00:00 2024-09-21T23:23:38.172 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 10/12 2024-09-21T23:23:38.339 INFO:teuthology.orchestra.run.smithi031.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.8 MB/s | 2.0 MB 00:00 2024-09-21T23:23:38.343 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:38.343 INFO:teuthology.orchestra.run.smithi031.stdout:Total 7.6 MB/s | 8.7 MB 00:01 2024-09-21T23:23:38.392 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:23:38.411 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:23:38.411 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:23:38.640 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:23:38.641 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:23:39.211 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:23:39.290 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 11/12 2024-09-21T23:23:39.298 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T23:23:39.323 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 12/12 2024-09-21T23:23:39.333 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T23:23:39.381 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T23:23:39.410 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T23:23:39.525 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T23:23:39.667 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T23:23:39.766 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T23:23:39.816 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T23:23:39.843 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T23:23:39.875 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 10/12 2024-09-21T23:23:40.656 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 12/12 2024-09-21T23:23:40.656 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 1/12 2024-09-21T23:23:40.656 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 2/12 2024-09-21T23:23:40.656 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 3/12 2024-09-21T23:23:40.656 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T23:23:40.656 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T23:23:40.656 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T23:23:40.656 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T23:23:40.656 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T23:23:40.656 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T23:23:40.657 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T23:23:40.657 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T23:23:40.982 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T23:23:40.982 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:40.982 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:23:40.982 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T23:23:40.982 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:40.982 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:40.982 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:40.982 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T23:23:40.983 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T23:23:40.983 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T23:23:40.983 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T23:23:40.983 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T23:23:40.983 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T23:23:40.983 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T23:23:40.983 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T23:23:40.983 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:40.983 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:23:40.994 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 11/12 2024-09-21T23:23:41.033 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 12/12 2024-09-21T23:23:41.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T23:23:41.821 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:06 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:23:41.960 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:23:41.961 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-09-21T23:23:41.961 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-21T23:23:41.961 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-09-21T23:23:41.961 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:23:41.961 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 7.4 M 2024-09-21T23:23:41.961 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-21T23:23:41.961 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T23:23:41.961 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:23:41.962 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-09-21T23:23:41.963 INFO:teuthology.orchestra.run.smithi060.stdout:Install 12 Packages 2024-09-21T23:23:41.963 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:41.963 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 38 M 2024-09-21T23:23:41.963 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 212 M 2024-09-21T23:23:41.963 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:23:42.235 INFO:teuthology.orchestra.run.smithi060.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 191 kB 00:00 2024-09-21T23:23:42.302 INFO:teuthology.orchestra.run.smithi060.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.2 MB/s | 801 kB 00:00 2024-09-21T23:23:42.327 INFO:teuthology.orchestra.run.smithi060.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 355 kB/s | 33 kB 00:00 2024-09-21T23:23:42.359 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 12/12 2024-09-21T23:23:42.359 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 1/12 2024-09-21T23:23:42.359 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 2/12 2024-09-21T23:23:42.359 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 3/12 2024-09-21T23:23:42.359 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T23:23:42.359 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T23:23:42.359 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T23:23:42.359 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T23:23:42.360 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T23:23:42.360 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T23:23:42.360 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T23:23:42.360 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T23:23:42.369 INFO:teuthology.orchestra.run.smithi060.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 441 kB/s | 18 kB 00:00 2024-09-21T23:23:42.412 INFO:teuthology.orchestra.run.smithi060.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 919 kB/s | 38 kB 00:00 2024-09-21T23:23:42.611 INFO:teuthology.orchestra.run.smithi060.stdout:(6/12): ceph-mgr-diskprediction-local-19.1.1-37 15 MB/s | 7.4 MB 00:00 2024-09-21T23:23:42.670 INFO:teuthology.orchestra.run.smithi060.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.1 MB/s | 3.0 MB 00:00 2024-09-21T23:23:42.684 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T23:23:42.684 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:23:42.684 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T23:23:42.684 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:42.684 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:42.684 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:42.684 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T23:23:42.685 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T23:23:42.685 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T23:23:42.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T23:23:42.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T23:23:42.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T23:23:42.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T23:23:42.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T23:23:42.685 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:23:42.745 INFO:teuthology.orchestra.run.smithi060.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2024-09-21T23:23:42.787 INFO:teuthology.orchestra.run.smithi060.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-21T23:23:42.863 INFO:teuthology.orchestra.run.smithi060.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 5.1 MB 00:00 2024-09-21T23:23:42.887 INFO:teuthology.orchestra.run.smithi060.stdout:(11/12): python3-devel-3.9.19-8.el9.x86_64.rpm 887 kB/s | 245 kB 00:00 2024-09-21T23:23:42.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T23:23:43.196 INFO:teuthology.orchestra.run.smithi060.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-09-21T23:23:43.197 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:43.197 INFO:teuthology.orchestra.run.smithi060.stdout:Total 31 MB/s | 38 MB 00:01 2024-09-21T23:23:43.540 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:07 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:23:43.560 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:23:43.589 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:23:43.589 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:23:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:23:43.684 INFO:teuthology.orchestra.run.smithi031.stdout:====================================================================================== 2024-09-21T23:23:43.684 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-21T23:23:43.684 INFO:teuthology.orchestra.run.smithi031.stdout:====================================================================================== 2024-09-21T23:23:43.684 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:23:43.684 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 7.4 M 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:43.685 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:23:43.686 INFO:teuthology.orchestra.run.smithi031.stdout:====================================================================================== 2024-09-21T23:23:43.686 INFO:teuthology.orchestra.run.smithi031.stdout:Install 12 Packages 2024-09-21T23:23:43.686 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:43.686 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 38 M 2024-09-21T23:23:43.686 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 212 M 2024-09-21T23:23:43.686 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:23:44.041 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:23:44.041 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:23:44.080 INFO:teuthology.orchestra.run.smithi031.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 848 kB/s | 191 kB 00:00 2024-09-21T23:23:44.147 INFO:teuthology.orchestra.run.smithi031.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.7 MB/s | 801 kB 00:00 2024-09-21T23:23:44.180 INFO:teuthology.orchestra.run.smithi031.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-09-21T23:23:44.231 INFO:teuthology.orchestra.run.smithi031.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-21T23:23:44.323 INFO:teuthology.orchestra.run.smithi031.stdout:(5/12): ceph-mgr-diskprediction-local-19.1.1-37 16 MB/s | 7.4 MB 00:00 2024-09-21T23:23:44.348 INFO:teuthology.orchestra.run.smithi031.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 329 kB/s | 38 kB 00:00 2024-09-21T23:23:44.482 INFO:teuthology.orchestra.run.smithi031.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-21T23:23:44.573 INFO:teuthology.orchestra.run.smithi031.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.0 MB/s | 3.0 MB 00:00 2024-09-21T23:23:44.623 INFO:teuthology.orchestra.run.smithi031.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.9 MB/s | 250 kB 00:00 2024-09-21T23:23:44.741 INFO:teuthology.orchestra.run.smithi031.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-09-21T23:23:44.824 INFO:teuthology.orchestra.run.smithi031.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 9.9 MB/s | 4.9 MB 00:00 2024-09-21T23:23:44.839 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:23:44.906 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T23:23:44.953 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T23:23:45.057 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T23:23:45.066 INFO:teuthology.orchestra.run.smithi031.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-09-21T23:23:45.067 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:45.067 INFO:teuthology.orchestra.run.smithi031.stdout:Total 28 MB/s | 38 MB 00:01 2024-09-21T23:23:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T23:23:45.287 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T23:23:45.328 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T23:23:45.407 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T23:23:45.430 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:23:45.459 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:23:45.459 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:23:45.931 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:23:45.931 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:23:46.097 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T23:23:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T23:23:46.721 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:23:46.793 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T23:23:46.840 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T23:23:46.944 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T23:23:46.982 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T23:23:47.174 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T23:23:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T23:23:47.293 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T23:23:47.549 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T23:23:47.977 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T23:23:48.106 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T23:23:48.638 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T23:23:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-21T23:23:49.426 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T23:23:50.500 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T23:23:50.530 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-21T23:23:51.180 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-21T23:23:51.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 1/12 2024-09-21T23:23:51.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T23:23:51.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T23:23:51.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T23:23:51.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T23:23:51.181 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T23:23:51.181 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T23:23:51.181 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T23:23:51.181 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T23:23:51.181 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T23:23:51.181 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T23:23:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T23:23:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:51.560 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:23:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T23:23:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T23:23:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T23:23:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T23:23:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T23:23:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T23:23:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T23:23:51.561 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T23:23:51.561 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T23:23:51.561 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T23:23:51.561 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T23:23:51.561 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:51.561 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:23:51.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-rook 2024-09-21T23:23:52.518 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:17 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:23:52.655 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 49 k 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T23:23:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T23:23:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T23:23:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T23:23:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T23:23:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T23:23:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:52.658 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:23:52.658 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:52.658 INFO:teuthology.orchestra.run.smithi060.stdout:Install 13 Packages 2024-09-21T23:23:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:52.659 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 2.2 M 2024-09-21T23:23:52.659 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 26 M 2024-09-21T23:23:52.659 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:23:52.828 INFO:teuthology.orchestra.run.smithi060.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 238 kB/s | 19 kB 00:00 2024-09-21T23:23:52.845 INFO:teuthology.orchestra.run.smithi060.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 263 kB/s | 26 kB 00:00 2024-09-21T23:23:52.903 INFO:teuthology.orchestra.run.smithi060.stdout:(3/13): ceph-mgr-rook-19.1.1-373.g4e45e6b4.el9. 316 kB/s | 49 kB 00:00 2024-09-21T23:23:52.929 INFO:teuthology.orchestra.run.smithi060.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-21T23:23:52.954 INFO:teuthology.orchestra.run.smithi060.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-21T23:23:52.971 INFO:teuthology.orchestra.run.smithi060.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.2 MB/s | 54 kB 00:00 2024-09-21T23:23:53.013 INFO:teuthology.orchestra.run.smithi060.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 551 kB/s | 32 kB 00:00 2024-09-21T23:23:53.030 INFO:teuthology.orchestra.run.smithi060.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 240 kB/s | 14 kB 00:00 2024-09-21T23:23:53.055 INFO:teuthology.orchestra.run.smithi060.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.8 MB/s | 279 kB 00:00 2024-09-21T23:23:53.082 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-21T23:23:53.082 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 1/12 2024-09-21T23:23:53.082 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T23:23:53.082 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T23:23:53.082 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T23:23:53.082 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T23:23:53.082 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T23:23:53.082 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T23:23:53.082 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T23:23:53.082 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T23:23:53.083 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T23:23:53.083 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T23:23:53.097 INFO:teuthology.orchestra.run.smithi060.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-21T23:23:53.139 INFO:teuthology.orchestra.run.smithi060.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 709 kB/s | 59 kB 00:00 2024-09-21T23:23:53.164 INFO:teuthology.orchestra.run.smithi060.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.4 MB/s | 90 kB 00:00 2024-09-21T23:23:53.214 INFO:teuthology.orchestra.run.smithi060.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-09-21T23:23:53.217 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:53.217 INFO:teuthology.orchestra.run.smithi060.stdout:Total 4.0 MB/s | 2.2 MB 00:00 2024-09-21T23:23:53.284 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:23:53.300 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:23:53.300 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:23:53.401 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:23:53.408 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:23:53.507 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T23:23:53.507 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T23:23:53.508 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T23:23:53.509 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:53.509 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:23:53.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-rook 2024-09-21T23:23:53.933 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:23:54.112 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T23:23:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T23:23:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T23:23:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T23:23:54.318 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T23:23:54.433 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T23:23:54.525 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:18 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:23:54.543 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T23:23:54.595 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T23:23:54.666 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:23:54.667 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-21T23:23:54.668 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:54.668 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:23:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 49 k 2024-09-21T23:23:54.668 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-21T23:23:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T23:23:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T23:23:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T23:23:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T23:23:54.668 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout:Install 13 Packages 2024-09-21T23:23:54.669 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:54.670 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.2 M 2024-09-21T23:23:54.670 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 26 M 2024-09-21T23:23:54.670 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:23:55.028 INFO:teuthology.orchestra.run.smithi031.stdout:(1/13): ceph-mgr-rook-19.1.1-373.g4e45e6b4.el9. 290 kB/s | 49 kB 00:00 2024-09-21T23:23:55.367 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T23:23:55.378 INFO:teuthology.orchestra.run.smithi031.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 37 kB/s | 19 kB 00:00 2024-09-21T23:23:55.420 INFO:teuthology.orchestra.run.smithi031.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 46 kB/s | 26 kB 00:00 2024-09-21T23:23:55.424 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T23:23:55.458 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T23:23:55.507 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T23:23:55.536 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-21T23:23:55.771 INFO:teuthology.orchestra.run.smithi031.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 299 kB/s | 222 kB 00:00 2024-09-21T23:23:56.079 INFO:teuthology.orchestra.run.smithi031.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 226 kB/s | 159 kB 00:00 2024-09-21T23:23:56.130 INFO:teuthology.orchestra.run.smithi031.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 660 kB/s | 32 kB 00:00 2024-09-21T23:23:56.146 INFO:teuthology.orchestra.run.smithi031.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 855 kB/s | 14 kB 00:00 2024-09-21T23:23:56.189 INFO:teuthology.orchestra.run.smithi031.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 363 kB/s | 279 kB 00:00 2024-09-21T23:23:56.230 INFO:teuthology.orchestra.run.smithi031.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 2.6 MB/s | 220 kB 00:00 2024-09-21T23:23:56.256 INFO:teuthology.orchestra.run.smithi031.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-09-21T23:23:56.306 INFO:teuthology.orchestra.run.smithi031.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-21T23:23:56.330 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-21T23:23:56.330 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 1/13 2024-09-21T23:23:56.330 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T23:23:56.330 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T23:23:56.330 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T23:23:56.330 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T23:23:56.331 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T23:23:56.331 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T23:23:56.331 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T23:23:56.331 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T23:23:56.331 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T23:23:56.331 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T23:23:56.331 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T23:23:56.390 INFO:teuthology.orchestra.run.smithi031.stdout:(12/13): python3-requests-oauthlib-1.3.0-12.el9 87 kB/s | 54 kB 00:00 2024-09-21T23:23:56.440 INFO:teuthology.orchestra.run.smithi031.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.1 MB/s | 1.0 MB 00:00 2024-09-21T23:23:56.444 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:56.444 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-09-21T23:23:56.517 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:23:56.535 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:23:56.535 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:23:56.632 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:23:56.634 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:23:56.658 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T23:23:56.658 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:56.658 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:23:56.658 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:56.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T23:23:56.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T23:23:56.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T23:23:56.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T23:23:56.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T23:23:56.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T23:23:56.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T23:23:56.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T23:23:56.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T23:23:56.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T23:23:56.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T23:23:56.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T23:23:56.659 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:56.659 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:23:56.930 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T23:23:57.166 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:23:57.354 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T23:23:57.410 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T23:23:57.474 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T23:23:57.509 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T23:23:57.542 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:22 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:23:57.560 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T23:23:57.675 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T23:23:57.680 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:23:57.681 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:57.681 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-21T23:23:57.681 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:57.681 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:23:57.681 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 146 k 2024-09-21T23:23:57.681 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-21T23:23:57.682 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T23:23:57.682 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T23:23:57.682 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T23:23:57.682 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T23:23:57.682 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T23:23:57.682 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:57.682 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:23:57.682 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:23:57.682 INFO:teuthology.orchestra.run.smithi060.stdout:Install 6 Packages 2024-09-21T23:23:57.682 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:23:57.683 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 7.0 M 2024-09-21T23:23:57.683 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 33 M 2024-09-21T23:23:57.683 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:23:57.785 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T23:23:57.837 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T23:23:58.050 INFO:teuthology.orchestra.run.smithi060.stdout:(1/6): ceph-mgr-cephadm-19.1.1-373.g4e45e6b4.el 706 kB/s | 146 kB 00:00 2024-09-21T23:23:58.142 INFO:teuthology.orchestra.run.smithi060.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.8 MB/s | 548 kB 00:00 2024-09-21T23:23:58.175 INFO:teuthology.orchestra.run.smithi060.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2024-09-21T23:23:58.201 INFO:teuthology.orchestra.run.smithi060.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-21T23:23:58.606 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T23:23:58.667 INFO:teuthology.orchestra.run.smithi060.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 301 kB/s | 248 kB 00:00 2024-09-21T23:23:58.674 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T23:23:58.708 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T23:23:58.757 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T23:23:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-21T23:23:59.134 INFO:teuthology.orchestra.run.smithi060.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2024-09-21T23:23:59.137 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:23:59.137 INFO:teuthology.orchestra.run.smithi060.stdout:Total 4.8 MB/s | 7.0 MB 00:01 2024-09-21T23:23:59.214 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:23:59.226 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:23:59.227 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:23:59.334 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:23:59.334 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:23:59.579 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-21T23:23:59.579 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 1/13 2024-09-21T23:23:59.579 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T23:23:59.579 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T23:23:59.580 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T23:23:59.580 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T23:23:59.580 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T23:23:59.580 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T23:23:59.580 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T23:23:59.580 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T23:23:59.580 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T23:23:59.580 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T23:23:59.580 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T23:23:59.642 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:23:59.817 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T23:23:59.879 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T23:23:59.932 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T23:23:59.932 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T23:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T23:23:59.934 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T23:23:59.934 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:23:59.934 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:00.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T23:24:00.435 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T23:24:00.526 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T23:24:00.597 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T23:24:00.623 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-21T23:24:00.809 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:24 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:00.949 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:00.949 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:00.949 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-21T23:24:00.949 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 146 k 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2024-09-21T23:24:00.950 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:00.951 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 7.0 M 2024-09-21T23:24:00.951 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 33 M 2024-09-21T23:24:00.951 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:24:01.266 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-21T23:24:01.300 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): ceph-mgr-cephadm-19.1.1-373.g4e45e6b4.el 619 kB/s | 146 kB 00:00 2024-09-21T23:24:01.358 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 993 kB/s | 58 kB 00:00 2024-09-21T23:24:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-21T23:24:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/6 2024-09-21T23:24:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T23:24:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T23:24:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T23:24:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T23:24:01.392 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-21T23:24:01.417 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.6 MB/s | 548 kB 00:00 2024-09-21T23:24:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T23:24:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:01.657 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:24:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:24:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T23:24:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T23:24:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T23:24:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T23:24:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T23:24:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:01.658 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:01.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-fuse 2024-09-21T23:24:01.917 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.0 MB/s | 6.0 MB 00:00 2024-09-21T23:24:01.920 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:01.920 INFO:teuthology.orchestra.run.smithi031.stdout:Total 7.2 MB/s | 7.0 MB 00:00 2024-09-21T23:24:01.993 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:24:02.005 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:24:02.005 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:24:02.111 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:24:02.112 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:24:02.431 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:24:02.513 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T23:24:02.652 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:02.653 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:02.653 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-21T23:24:02.653 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:02.653 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:24:02.653 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 897 k 2024-09-21T23:24:02.653 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-21T23:24:02.653 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T23:24:02.653 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:02.653 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:24:02.653 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:02.653 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2024-09-21T23:24:02.654 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:02.654 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 976 k 2024-09-21T23:24:02.654 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 2.9 M 2024-09-21T23:24:02.654 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:24:02.661 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T23:24:02.978 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): ceph-fuse-19.1.1-373.g4e45e6b4.el9.x86_6 3.3 MB/s | 897 kB 00:00 2024-09-21T23:24:03.170 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 173 kB/s | 80 kB 00:00 2024-09-21T23:24:03.170 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:03.171 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.8 MB/s | 976 kB 00:00 2024-09-21T23:24:03.183 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:24:03.206 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:24:03.206 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:24:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T23:24:03.293 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:24:03.293 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:24:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T23:24:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T23:24:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-21T23:24:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:24:03.626 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T23:24:03.697 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-21T23:24:04.197 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-21T23:24:04.197 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/6 2024-09-21T23:24:04.197 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T23:24:04.197 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T23:24:04.197 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T23:24:04.198 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T23:24:04.438 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-21T23:24:04.438 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/2 2024-09-21T23:24:04.472 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T23:24:04.472 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:04.472 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:24:04.472 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:24:04.472 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T23:24:04.472 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T23:24:04.472 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T23:24:04.472 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T23:24:04.472 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T23:24:04.472 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:04.473 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:04.722 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2024-09-21T23:24:04.740 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T23:24:04.740 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:04.740 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:24:04.740 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T23:24:04.740 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:04.740 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:05.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-volume 2024-09-21T23:24:05.325 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:29 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:05.464 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:05.465 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-21T23:24:05.465 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:05.465 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:24:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 897 k 2024-09-21T23:24:05.465 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-21T23:24:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T23:24:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:05.465 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:24:05.465 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:05.465 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2024-09-21T23:24:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:05.466 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 976 k 2024-09-21T23:24:05.466 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 2.9 M 2024-09-21T23:24:05.466 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:24:05.614 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:05.618 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 608 kB/s | 80 kB 00:00 2024-09-21T23:24:05.753 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:05.753 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-21T23:24:05.753 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:05.753 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:24:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 285 k 2024-09-21T23:24:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:05.754 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:24:05.754 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:05.754 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-21T23:24:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:05.754 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 285 k 2024-09-21T23:24:05.754 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.3 M 2024-09-21T23:24:05.754 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:24:05.819 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-19.1.1-373.g4e45e6b4.el9.x86_6 2.6 MB/s | 897 kB 00:00 2024-09-21T23:24:05.819 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:05.820 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.7 MB/s | 976 kB 00:00 2024-09-21T23:24:05.832 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:24:05.855 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:24:05.855 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:24:05.942 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:24:05.942 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:24:05.982 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-volume-19.1.1-373.g4e45e6b4.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-21T23:24:05.983 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:05.983 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-21T23:24:05.983 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:24:05.993 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:24:05.993 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:24:06.020 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:24:06.020 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:24:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:24:06.254 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:24:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T23:24:06.334 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:24:06.352 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:24:06.352 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:24:06.352 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T23:24:06.352 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-21T23:24:07.038 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-21T23:24:07.039 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/2 2024-09-21T23:24:07.313 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T23:24:07.313 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:07.313 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:24:07.313 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T23:24:07.313 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:07.314 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:07.395 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:24:07.395 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:07.395 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:24:07.395 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:24:07.395 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:07.395 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:07.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-volume 2024-09-21T23:24:07.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados-devel 2024-09-21T23:24:08.192 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:08.310 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:08.332 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:08.332 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:08.332 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-21T23:24:08.332 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:08.332 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:24:08.332 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 285 k 2024-09-21T23:24:08.333 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:08.333 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:24:08.333 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:08.333 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-21T23:24:08.333 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:08.333 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 285 k 2024-09-21T23:24:08.333 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.3 M 2024-09-21T23:24:08.334 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:24:08.450 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:08.451 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:08.451 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-09-21T23:24:08.451 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:08.451 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:24:08.451 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 127 k 2024-09-21T23:24:08.451 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:08.451 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:24:08.451 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:08.452 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-21T23:24:08.452 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:08.452 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 127 k 2024-09-21T23:24:08.452 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 456 k 2024-09-21T23:24:08.452 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:24:08.556 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-volume-19.1.1-373.g4e45e6b4.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-21T23:24:08.557 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:08.557 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-21T23:24:08.557 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:24:08.567 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:24:08.567 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:24:08.600 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:24:08.601 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:24:08.637 INFO:teuthology.orchestra.run.smithi060.stdout:librados-devel-19.1.1-373.g4e45e6b4.el9.x86_64. 684 kB/s | 127 kB 00:00 2024-09-21T23:24:08.638 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:08.638 INFO:teuthology.orchestra.run.smithi060.stdout:Total 679 kB/s | 127 kB 00:00 2024-09-21T23:24:08.638 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:24:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:24:08.644 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:24:08.711 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:24:08.712 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:24:08.835 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:24:08.872 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:24:08.900 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:24:08.917 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:24:08.917 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:24:08.918 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T23:24:08.918 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:08.980 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:09.437 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:09.682 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:09.682 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:09.683 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:24:09.683 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:24:09.683 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:09.683 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:09.915 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs2 2024-09-21T23:24:09.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-21T23:24:09.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:09.979 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:24:09.979 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T23:24:09.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:09.979 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:10.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2024-09-21T23:24:10.522 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:10.616 INFO:teuthology.orchestra.run.smithi060.stdout:Package libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:10.662 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:10.663 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-21T23:24:10.663 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:10.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs-devel 2024-09-21T23:24:10.892 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:11.032 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:11.032 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:11.032 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-09-21T23:24:11.032 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:11.032 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:24:11.032 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 127 k 2024-09-21T23:24:11.032 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:11.033 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:24:11.033 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:11.033 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-21T23:24:11.033 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:11.033 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 127 k 2024-09-21T23:24:11.033 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 456 k 2024-09-21T23:24:11.033 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:24:11.244 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-19.1.1-373.g4e45e6b4.el9.x86_64. 602 kB/s | 127 kB 00:00 2024-09-21T23:24:11.245 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:11.245 INFO:teuthology.orchestra.run.smithi031.stdout:Total 597 kB/s | 127 kB 00:00 2024-09-21T23:24:11.245 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:24:11.251 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:24:11.251 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:24:11.319 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:24:11.319 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:24:11.339 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:11.479 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:11.480 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:11.480 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-09-21T23:24:11.480 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:11.480 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:24:11.480 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 32 k 2024-09-21T23:24:11.480 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:11.480 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:24:11.480 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:11.480 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-21T23:24:11.480 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:11.481 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:24:11.481 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 32 k 2024-09-21T23:24:11.481 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 142 k 2024-09-21T23:24:11.481 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:24:11.604 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:11.632 INFO:teuthology.orchestra.run.smithi060.stdout:libcephfs-devel-19.1.1-373.g4e45e6b4.el9.x86_64 210 kB/s | 32 kB 00:00 2024-09-21T23:24:11.633 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:11.633 INFO:teuthology.orchestra.run.smithi060.stdout:Total 208 kB/s | 32 kB 00:00 2024-09-21T23:24:11.633 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:24:11.637 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:24:11.637 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:24:11.648 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:24:11.648 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:24:11.741 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:24:11.850 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:12.027 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:12.213 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:12.288 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:12.289 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:12.289 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:24:12.289 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:24:12.289 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:12.289 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:12.454 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:12.455 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:12.455 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:24:12.455 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:24:12.455 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:12.455 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:12.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2024-09-21T23:24:12.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados2 2024-09-21T23:24:13.121 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:13.213 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:13.242 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:13.259 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:13.260 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-21T23:24:13.261 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:13.334 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:13.340 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2024-09-21T23:24:13.380 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:13.381 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-21T23:24:13.381 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:13.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librbd1 2024-09-21T23:24:13.965 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:14.057 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:14.109 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 32 k 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 32 k 2024-09-21T23:24:14.110 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 142 k 2024-09-21T23:24:14.111 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:24:14.149 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:14.196 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:14.197 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-21T23:24:14.197 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:14.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rados 2024-09-21T23:24:14.265 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-19.1.1-373.g4e45e6b4.el9.x86_64 206 kB/s | 32 kB 00:00 2024-09-21T23:24:14.266 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:14.266 INFO:teuthology.orchestra.run.smithi031.stdout:Total 203 kB/s | 32 kB 00:00 2024-09-21T23:24:14.266 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:24:14.270 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:24:14.270 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:24:14.286 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:24:14.287 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:24:14.381 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:24:14.499 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:14.865 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:14.867 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:14.959 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:15.005 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:15.006 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-21T23:24:15.006 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:15.072 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rgw 2024-09-21T23:24:15.094 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:15.094 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:15.094 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:24:15.094 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:24:15.094 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:15.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2024-09-21T23:24:15.679 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:15.771 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:15.817 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:15.819 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-21T23:24:15.819 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:15.878 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:15.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-cephfs 2024-09-21T23:24:15.972 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:16.018 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:16.019 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-21T23:24:16.019 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:16.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2024-09-21T23:24:16.494 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:16.586 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:16.632 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:16.633 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-21T23:24:16.633 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:16.695 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:16.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rbd 2024-09-21T23:24:16.788 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:16.834 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:16.835 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-21T23:24:16.835 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:16.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2024-09-21T23:24:17.297 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:42 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:17.388 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:17.434 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:17.435 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-21T23:24:17.435 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:17.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-fuse 2024-09-21T23:24:17.508 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:17.600 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:17.646 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:17.647 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-21T23:24:17.647 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:17.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2024-09-21T23:24:18.103 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:18.241 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:18.242 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:18.242 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-09-21T23:24:18.242 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:18.242 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:24:18.242 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 85 k 2024-09-21T23:24:18.242 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:18.242 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:24:18.243 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:18.243 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-21T23:24:18.243 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:18.243 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 85 k 2024-09-21T23:24:18.243 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 231 k 2024-09-21T23:24:18.243 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:24:18.321 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:42 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:18.411 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-fuse-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 510 kB/s | 85 kB 00:00 2024-09-21T23:24:18.411 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:18.411 INFO:teuthology.orchestra.run.smithi060.stdout:Total 506 kB/s | 85 kB 00:00 2024-09-21T23:24:18.412 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:24:18.412 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:18.419 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:24:18.419 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:24:18.459 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:18.460 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-21T23:24:18.460 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:18.486 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:24:18.486 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:24:18.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2024-09-21T23:24:18.649 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:24:18.769 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:19.141 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:19.145 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:19.239 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:19.286 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:19.287 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-21T23:24:19.287 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:19.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2024-09-21T23:24:19.430 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:19.430 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:19.430 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:24:19.430 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:24:19.430 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:19.430 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:19.698 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-mirror 2024-09-21T23:24:19.960 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:20.052 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T23:24:20.098 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:20.099 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-21T23:24:20.099 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:20.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2024-09-21T23:24:20.317 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:45 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:20.456 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:20.457 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:20.457 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-09-21T23:24:20.457 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:20.457 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:24:20.457 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-21T23:24:20.457 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:20.457 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:24:20.457 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:20.457 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-21T23:24:20.457 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:20.458 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 3.1 M 2024-09-21T23:24:20.458 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 13 M 2024-09-21T23:24:20.458 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:24:20.779 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:44 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:20.797 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-mirror-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 9.1 MB/s | 3.1 MB 00:00 2024-09-21T23:24:20.798 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:20.799 INFO:teuthology.orchestra.run.smithi060.stdout:Total 9.1 MB/s | 3.1 MB 00:00 2024-09-21T23:24:20.799 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:24:20.809 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:24:20.809 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:24:20.895 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:24:20.896 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:24:20.916 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:20.917 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:20.917 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-09-21T23:24:20.917 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:20.917 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:24:20.917 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 85 k 2024-09-21T23:24:20.917 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:20.917 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:24:20.917 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:20.917 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-21T23:24:20.918 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:20.918 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 85 k 2024-09-21T23:24:20.918 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 231 k 2024-09-21T23:24:20.918 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:24:21.117 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 429 kB/s | 85 kB 00:00 2024-09-21T23:24:21.118 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:21.118 INFO:teuthology.orchestra.run.smithi031.stdout:Total 426 kB/s | 85 kB 00:00 2024-09-21T23:24:21.118 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:24:21.126 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:24:21.126 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:24:21.194 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:24:21.194 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:24:21.346 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:24:21.352 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:24:21.415 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:21.443 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:21.443 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:24:21.443 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T23:24:21.443 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T23:24:21.443 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T23:24:21.443 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:21.451 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:21.837 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:22.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:22.103 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:22.104 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:24:22.104 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:24:22.104 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:22.104 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:22.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2024-09-21T23:24:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:22.626 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:24:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:24:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:22.626 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:22.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-nbd 2024-09-21T23:24:22.985 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:23.124 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:23.125 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-09-21T23:24:23.125 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:23.125 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:24:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-21T23:24:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:23.125 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:24:23.125 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:23.125 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-21T23:24:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:23.126 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.1 M 2024-09-21T23:24:23.126 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 13 M 2024-09-21T23:24:23.126 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:24:23.462 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 9.3 MB/s | 3.1 MB 00:00 2024-09-21T23:24:23.462 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:23.463 INFO:teuthology.orchestra.run.smithi031.stdout:Total 9.2 MB/s | 3.1 MB 00:00 2024-09-21T23:24:23.463 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:24:23.472 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:24:23.473 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:24:23.474 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:48 ago on Sat 21 Sep 2024 11:22:35 PM UTC. 2024-09-21T23:24:23.558 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:24:23.558 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:24:23.615 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-21T23:24:23.615 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:23.616 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-09-21T23:24:23.616 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:23.616 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-21T23:24:23.616 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 170 k 2024-09-21T23:24:23.616 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:23.616 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-21T23:24:23.616 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-21T23:24:23.616 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-21T23:24:23.616 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:23.617 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 170 k 2024-09-21T23:24:23.617 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 494 k 2024-09-21T23:24:23.617 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-21T23:24:23.826 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-nbd-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 815 kB/s | 170 kB 00:00 2024-09-21T23:24:23.827 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:23.827 INFO:teuthology.orchestra.run.smithi060.stdout:Total 809 kB/s | 170 kB 00:00 2024-09-21T23:24:23.827 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-21T23:24:23.836 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-21T23:24:23.836 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-21T23:24:23.904 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-21T23:24:23.904 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-21T23:24:24.013 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:24:24.066 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-21T23:24:24.070 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:24.097 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:24.097 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T23:24:24.097 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T23:24:24.097 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T23:24:24.097 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T23:24:24.097 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:24.176 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:24.586 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:24.840 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:24.841 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:24.841 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-21T23:24:24.841 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:24:24.841 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:24.841 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-21T23:24:25.068 DEBUG:teuthology.parallel:result is None 2024-09-21T23:24:25.235 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:25.235 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:25.235 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:24:25.236 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:24:25.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:25.236 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:25.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2024-09-21T23:24:26.076 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:50 ago on Sat 21 Sep 2024 11:22:36 PM UTC. 2024-09-21T23:24:26.215 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-21T23:24:26.216 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:26.216 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-09-21T23:24:26.216 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:26.216 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-21T23:24:26.217 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 170 k 2024-09-21T23:24:26.217 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:26.217 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-21T23:24:26.217 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-09-21T23:24:26.217 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-09-21T23:24:26.217 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:26.217 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 170 k 2024-09-21T23:24:26.217 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 494 k 2024-09-21T23:24:26.218 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-21T23:24:26.432 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 791 kB/s | 170 kB 00:00 2024-09-21T23:24:26.433 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-21T23:24:26.433 INFO:teuthology.orchestra.run.smithi031.stdout:Total 786 kB/s | 170 kB 00:00 2024-09-21T23:24:26.433 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-21T23:24:26.441 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-21T23:24:26.441 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-21T23:24:26.509 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-21T23:24:26.509 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-21T23:24:26.671 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-21T23:24:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T23:24:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:27.457 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-21T23:24:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T23:24:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:27.457 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-21T23:24:27.687 DEBUG:teuthology.parallel:result is None 2024-09-21T23:24:27.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:24:27.833 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T23:24:27.855 INFO:teuthology.orchestra.run.smithi031.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-21T23:24:27.855 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373.g4e45e6b4.el9 2024-09-21T23:24:27.855 INFO:teuthology.task.install:The correct ceph version 19.1.1-373.g4e45e6b4 is installed. 2024-09-21T23:24:27.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:24:27.992 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T23:24:28.018 INFO:teuthology.orchestra.run.smithi060.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-21T23:24:28.018 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373.g4e45e6b4.el9 2024-09-21T23:24:28.018 INFO:teuthology.task.install:The correct ceph version 19.1.1-373.g4e45e6b4 is installed. 2024-09-21T23:24:28.020 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T23:24:28.020 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:24:28.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T23:24:28.047 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:24:28.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T23:24:28.088 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T23:24:28.088 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:24:28.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T23:24:28.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T23:24:28.179 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:24:28.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T23:24:28.208 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T23:24:28.275 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T23:24:28.276 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:24:28.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T23:24:28.302 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T23:24:28.368 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:24:28.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T23:24:28.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T23:24:28.463 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T23:24:28.464 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:24:28.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T23:24:28.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T23:24:28.555 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:24:28.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T23:24:28.584 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T23:24:28.653 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T23:24:28.726 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-21T23:24:28.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2024-09-21T23:24:28.728 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2024-09-21T23:24:28.756 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-21T23:24:28.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T23:24:28.794 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T23:24:28.826 INFO:tasks.ceph:Creating extra log directories... 2024-09-21T23:24:28.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T23:24:28.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T23:24:28.903 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-21T23:24:28.903 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-21T23:24:28.903 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523', 'branch': 'squid', 'description': 'rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/journal}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914523', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 1478, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'sleep_before_teardown': 0, 'subset': '24/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjsKFqN5SC1/+BMa9J8b+HKXSYZR3yPUgK33tyW4AfS/ctwQYgeq+4ts56rDc59R96WoIpPs4k2sOxuhBEvtA4=', 'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5BtSD7dQLokjnkXce81rFE2SMz4ZZ0SctCFcIbe1CfQX6xWVjh/o1itejmj+PIvv4PgsqC8P18bmGK6lFugnE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rbd/journal.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T23:24:28.904 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T23:24:28.926 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T23:24:28.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T23:24:28.981 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T23:24:29.019 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:24:29.019 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T23:24:29.060 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T23:24:29.060 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-09-21T23:24:29.116 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T23:24:29.116 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:24:29.116 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-21T23:24:29.116 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:24:29.116 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T23:24:29.116 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-21 23:24:24.356371267 +0000 2024-09-21T23:24:29.116 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-21 23:21:54.920012945 +0000 2024-09-21T23:24:29.117 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-21 23:21:54.920012945 +0000 2024-09-21T23:24:29.117 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-21 23:21:54.920012945 +0000 2024-09-21T23:24:29.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T23:24:29.180 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-21T23:24:29.180 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-21T23:24:29.181 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000201618 s, 2.5 MB/s 2024-09-21T23:24:29.182 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T23:24:29.239 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-09-21T23:24:29.295 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T23:24:29.295 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:24:29.296 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T23:24:29.296 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:24:29.296 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T23:24:29.296 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-21 23:24:24.357371251 +0000 2024-09-21T23:24:29.296 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-21 23:21:55.276007234 +0000 2024-09-21T23:24:29.296 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-21 23:21:55.276007234 +0000 2024-09-21T23:24:29.296 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-21 23:21:55.276007234 +0000 2024-09-21T23:24:29.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T23:24:29.360 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-21T23:24:29.360 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-21T23:24:29.360 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000200942 s, 2.5 MB/s 2024-09-21T23:24:29.361 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T23:24:29.418 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-09-21T23:24:29.474 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T23:24:29.474 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:24:29.474 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-21T23:24:29.475 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:24:29.475 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T23:24:29.475 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-21 23:24:24.357371251 +0000 2024-09-21T23:24:29.475 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-21 23:21:55.604001972 +0000 2024-09-21T23:24:29.475 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-21 23:21:55.604001972 +0000 2024-09-21T23:24:29.475 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-21 23:21:55.604001972 +0000 2024-09-21T23:24:29.475 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T23:24:29.538 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-21T23:24:29.538 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-21T23:24:29.538 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000163442 s, 3.1 MB/s 2024-09-21T23:24:29.540 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T23:24:29.597 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-09-21T23:24:29.653 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T23:24:29.653 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:24:29.653 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-21T23:24:29.653 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:24:29.653 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T23:24:29.654 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-21 23:24:24.357371251 +0000 2024-09-21T23:24:29.654 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-21 23:21:55.947996454 +0000 2024-09-21T23:24:29.654 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-21 23:21:55.947996454 +0000 2024-09-21T23:24:29.654 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-21 23:21:55.947996454 +0000 2024-09-21T23:24:29.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T23:24:29.717 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-21T23:24:29.717 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-21T23:24:29.717 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000229157 s, 2.2 MB/s 2024-09-21T23:24:29.718 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T23:24:29.775 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T23:24:29.775 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:24:29.775 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T23:24:29.795 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T23:24:29.795 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-09-21T23:24:29.854 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T23:24:29.854 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:24:29.854 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-21T23:24:29.854 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:24:29.854 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T23:24:29.854 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-21 23:24:21.729044567 +0000 2024-09-21T23:24:29.854 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-21 23:21:54.894839207 +0000 2024-09-21T23:24:29.854 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-21 23:21:54.894839207 +0000 2024-09-21T23:24:29.854 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-21 23:21:54.894839207 +0000 2024-09-21T23:24:29.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T23:24:29.923 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-21T23:24:29.923 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-21T23:24:29.923 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000104741 s, 4.9 MB/s 2024-09-21T23:24:29.924 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T23:24:29.984 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-09-21T23:24:30.044 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T23:24:30.044 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:24:30.044 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-21T23:24:30.044 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:24:30.044 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T23:24:30.044 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-21 23:24:21.729044567 +0000 2024-09-21T23:24:30.044 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-21 23:21:55.223836192 +0000 2024-09-21T23:24:30.044 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-21 23:21:55.223836192 +0000 2024-09-21T23:24:30.045 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-21 23:21:55.223836192 +0000 2024-09-21T23:24:30.045 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T23:24:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-21T23:24:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-21T23:24:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000118467 s, 4.3 MB/s 2024-09-21T23:24:30.114 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T23:24:30.174 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-09-21T23:24:30.232 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T23:24:30.233 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:24:30.233 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-21T23:24:30.233 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:24:30.233 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T23:24:30.233 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-21 23:24:21.730044558 +0000 2024-09-21T23:24:30.233 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-21 23:21:55.537833314 +0000 2024-09-21T23:24:30.233 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-21 23:21:55.537833314 +0000 2024-09-21T23:24:30.233 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-21 23:21:55.537833314 +0000 2024-09-21T23:24:30.233 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T23:24:30.299 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-21T23:24:30.299 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-21T23:24:30.299 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.2741e-05 s, 5.5 MB/s 2024-09-21T23:24:30.300 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T23:24:30.358 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-09-21T23:24:30.416 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T23:24:30.416 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:24:30.416 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-21T23:24:30.416 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:24:30.416 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T23:24:30.416 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-21 23:24:21.730044558 +0000 2024-09-21T23:24:30.416 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-21 23:21:55.879830179 +0000 2024-09-21T23:24:30.416 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-21 23:21:55.879830179 +0000 2024-09-21T23:24:30.417 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-21 23:21:55.879830179 +0000 2024-09-21T23:24:30.417 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T23:24:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-21T23:24:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-21T23:24:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000108076 s, 4.7 MB/s 2024-09-21T23:24:30.483 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T23:24:30.540 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-21T23:24:30.541 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-21T23:24:30.541 INFO:tasks.ceph:Generating config... 2024-09-21T23:24:30.542 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-21T23:24:30.542 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-21T23:24:30.542 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-21T23:24:30.542 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T23:24:30.542 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T23:24:30.542 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-21T23:24:30.542 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-21T23:24:30.542 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-21T23:24:30.542 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-21T23:24:30.542 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-21T23:24:30.543 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-21T23:24:30.544 INFO:tasks.ceph:Setting up mon.a... 2024-09-21T23:24:30.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-21T23:24:30.603 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.keyring 2024-09-21T23:24:30.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-21T23:24:30.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T23:24:30.679 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.31'), ('mon.c', '[v2:172.21.15.31:3301,v1:172.21.15.31:6790]'), ('mon.b', '172.21.15.60')] 2024-09-21T23:24:30.680 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31,[v2:172.21.15.31:3301,v1:172.21.15.31:6790],172.21.15.60', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-21T23:24:30.681 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:24:30.682 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T23:24:30.735 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --addv c '[v2:172.21.15.31:3301,v1:172.21.15.31:6790]' --add b 172.21.15.60 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid d51cc561-ddb8-41ef-b61c-7c6d698e4dd0 2024-09-21T23:24:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = quincy 2024-09-21T23:24:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2024-09-21T23:24:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:fsid d51cc561-ddb8-41ef-b61c-7c6d698e4dd0 2024-09-21T23:24:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2024-09-21T23:24:30.823656+0000 2024-09-21T23:24:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-09-21T23:24:30.823656+0000 2024-09-21T23:24:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 17 (quincy) 2024-09-21T23:24:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2024-09-21T23:24:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-09-21T23:24:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:1: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-09-21T23:24:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:2: [v2:172.21.15.31:3301/0,v1:172.21.15.31:6790/0] mon.c 2024-09-21T23:24:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-21T23:24:30.825 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T23:24:30.840 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d51cc561-ddb8-41ef-b61c-7c6d698e4dd0... 2024-09-21T23:24:30.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T23:24:30.882 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.925 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2024-09-21T23:24:30.926 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2024-09-21T23:24:30.927 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = "172.21.15.31,[v2:172.21.15.31:3301,v1:172.21.15.31:6790],172.21.15.60" 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on pool no app = False 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 5000 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = d51cc561-ddb8-41ef-b61c-7c6d698e4dd0 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.928 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug trim objects = true 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2024-09-21T23:24:30.929 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore compression algorithm = lz4 2024-09-21T23:24:30.930 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore compression mode = aggressive 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 1/20 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 1/20 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 4/10 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: osd blocked scrub grace period = 3600 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: osd max markdown count = 1000 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2024-09-21T23:24:30.931 INFO:teuthology.orchestra.run.smithi031.stdout: osd shutdown pgref assert = True 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-21T23:24:30.932 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T23:24:30.933 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T23:24:30.934 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2024-09-21T23:24:30.935 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.c] 2024-09-21T23:24:30.935 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.b] 2024-09-21T23:24:30.935 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-09-21T23:24:30.935 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2024-09-21T23:24:30.935 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T23:24:30.935 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2024-09-21T23:24:30.935 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.935 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2024-09-21T23:24:30.935 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.936 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T23:24:30.937 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = "172.21.15.31,[v2:172.21.15.31:3301,v1:172.21.15.31:6790],172.21.15.60" 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: mon client directed command retry = 5 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on pool no app = False 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject socket failures = 5000 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = d51cc561-ddb8-41ef-b61c-7c6d698e4dd0 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2024-09-21T23:24:30.938 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug trim objects = true 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2024-09-21T23:24:30.939 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2024-09-21T23:24:30.940 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore compression algorithm = lz4 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore compression mode = aggressive 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 1/20 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 1/20 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 4/10 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: osd blocked scrub grace period = 3600 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: osd max markdown count = 1000 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2024-09-21T23:24:30.941 INFO:teuthology.orchestra.run.smithi060.stdout: osd shutdown pgref assert = True 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-21T23:24:30.942 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T23:24:30.943 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T23:24:30.944 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2024-09-21T23:24:30.945 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.c] 2024-09-21T23:24:30.945 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.b] 2024-09-21T23:24:30.946 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-21T23:24:30.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-21T23:24:31.040 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T23:24:31.041 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:24:31.041 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-21T23:24:31.056 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:24:31.056 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-21T23:24:31.111 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:24:31.112 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:24:31.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T23:24:31.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T23:24:31.186 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:24:31.186 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:31.242 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:24:31.242 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:24:31.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T23:24:31.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T23:24:31.285 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:24:31.285 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:31.344 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T23:24:31.344 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T23:24:31.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-21T23:24:31.396 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-21T23:24:31.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T23:24:31.493 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T23:24:31.496 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T23:24:31.496 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T23:24:31.496 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T23:24:31.539 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-21T23:24:31.552 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T23:24:31.598 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-21T23:24:31.612 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T23:24:31.612 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-21T23:24:31.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-21T23:24:31.638 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T23:24:31.638 INFO:tasks.ceph:role: osd.0 2024-09-21T23:24:31.638 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:24:31.638 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T23:24:31.710 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:24:31.711 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:24:31.711 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:24:31.711 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T23:24:31.711 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:24:31.711 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:24:31.711 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:24:31.711 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T23:24:31.711 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:24:31.711 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:24:32.219 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-21T23:24:32.221 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-21T23:24:32.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-21T23:24:32.322 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-21T23:24:32.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-21T23:24:32.423 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T23:24:32.423 INFO:tasks.ceph:role: osd.1 2024-09-21T23:24:32.423 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:24:32.423 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T23:24:32.496 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:24:32.496 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:24:32.496 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:24:32.497 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T23:24:32.497 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:24:32.497 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:24:32.497 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:24:32.497 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T23:24:32.497 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:24:32.497 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:24:32.962 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-21T23:24:32.964 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-21T23:24:32.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-21T23:24:33.046 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-21T23:24:33.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-21T23:24:33.183 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T23:24:33.183 INFO:tasks.ceph:role: osd.2 2024-09-21T23:24:33.184 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:24:33.184 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T23:24:33.255 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:24:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:24:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:24:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T23:24:33.256 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:24:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:24:33.256 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:24:33.256 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T23:24:33.256 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:24:33.256 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:24:33.766 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-21T23:24:33.768 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-21T23:24:33.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-21T23:24:33.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-21T23:24:33.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-21T23:24:34.003 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T23:24:34.003 INFO:tasks.ceph:role: osd.3 2024-09-21T23:24:34.003 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:24:34.003 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-21T23:24:34.076 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:24:34.076 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:24:34.076 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:24:34.076 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T23:24:34.076 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:24:34.076 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:24:34.076 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:24:34.077 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T23:24:34.077 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:24:34.077 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:24:34.589 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-21T23:24:34.591 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-21T23:24:34.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-21T23:24:34.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-21T23:24:34.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:34.828+0000 7f09f4483640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-21T23:24:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:34.828+0000 7f09f4483640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-21T23:24:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:34.828+0000 7f09f4483640 -1 bdev(0x55d4f20d5800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-21T23:24:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:34.828+0000 7f09f4483640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-21T23:24:38.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T23:24:38.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:38.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:38.362+0000 7fdbf3412640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-21T23:24:38.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:38.363+0000 7fdbf3412640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-21T23:24:38.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:38.363+0000 7fdbf3412640 -1 bdev(0x559774fd7800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-21T23:24:38.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:38.363+0000 7fdbf3412640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-21T23:24:41.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T23:24:41.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:41.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:41.652+0000 7f36318e2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-21T23:24:41.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:41.652+0000 7f36318e2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-21T23:24:41.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:41.652+0000 7f36318e2640 -1 bdev(0x5601e779f800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-21T23:24:41.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:41.652+0000 7f36318e2640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-21T23:24:44.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T23:24:44.598 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:44.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:44.688+0000 7ff19dc14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-21T23:24:44.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:44.688+0000 7ff19dc14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-21T23:24:44.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:44.688+0000 7ff19dc14640 -1 bdev(0x55e0a0f2f800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-21T23:24:44.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:24:44.688+0000 7ff19dc14640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-21T23:24:47.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T23:24:47.632 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-21T23:24:47.663 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-21T23:24:47.663 INFO:tasks.ceph:role: osd.4 2024-09-21T23:24:47.663 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:24:47.664 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T23:24:47.733 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:24:47.733 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:24:47.733 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:24:47.733 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T23:24:47.733 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:24:47.733 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:24:47.733 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:24:47.733 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T23:24:47.733 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:24:47.733 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:24:47.767 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-09-21T23:24:47.768 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-21T23:24:47.769 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-21T23:24:47.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-21T23:24:47.912 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-21T23:24:47.979 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-21T23:24:47.979 INFO:tasks.ceph:role: osd.5 2024-09-21T23:24:47.979 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:24:47.979 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T23:24:48.045 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:24:48.045 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:24:48.045 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:24:48.045 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T23:24:48.045 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:24:48.045 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:24:48.045 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:24:48.045 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T23:24:48.045 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:24:48.045 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:24:48.077 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-09-21T23:24:48.079 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-21T23:24:48.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-21T23:24:48.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-21T23:24:48.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-21T23:24:48.287 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-21T23:24:48.287 INFO:tasks.ceph:role: osd.6 2024-09-21T23:24:48.287 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:24:48.287 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T23:24:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:24:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:24:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:24:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T23:24:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:24:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:24:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:24:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T23:24:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:24:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:24:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-09-21T23:24:48.389 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-21T23:24:48.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-21T23:24:48.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-21T23:24:48.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-21T23:24:48.595 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-21T23:24:48.595 INFO:tasks.ceph:role: osd.7 2024-09-21T23:24:48.595 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:24:48.595 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-21T23:24:48.660 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:24:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:24:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:24:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T23:24:48.661 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:24:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:24:48.661 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:24:48.661 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T23:24:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:24:48.661 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:24:48.692 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-09-21T23:24:48.694 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-21T23:24:48.695 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-21T23:24:48.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-21T23:24:48.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:48.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:48.923+0000 7f52e1216640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-21T23:24:48.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:48.923+0000 7f52e1216640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-21T23:24:48.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:48.923+0000 7f52e1216640 -1 bdev(0x55cb5c915800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-21T23:24:48.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:48.923+0000 7f52e1216640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-21T23:24:51.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-21T23:24:51.873 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:51.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:51.975+0000 7f203080d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-21T23:24:51.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:51.975+0000 7f203080d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-21T23:24:51.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:51.975+0000 7f203080d640 -1 bdev(0x55c30ec65800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-21T23:24:51.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:51.975+0000 7f203080d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-21T23:24:55.379 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T23:24:55.410 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:55.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:55.502+0000 7f3252f20640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-21T23:24:55.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:55.502+0000 7f3252f20640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-21T23:24:55.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:55.503+0000 7f3252f20640 -1 bdev(0x5598bc7f5800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-21T23:24:55.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:55.503+0000 7f3252f20640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-21T23:24:58.413 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T23:24:58.443 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:24:58.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:58.534+0000 7fd89ec2c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-21T23:24:58.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:58.534+0000 7fd89ec2c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-21T23:24:58.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:58.534+0000 7fd89ec2c640 -1 bdev(0x560d8d9ff800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-21T23:24:58.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:24:58.534+0000 7fd89ec2c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-21T23:25:01.698 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T23:25:01.728 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T23:25:01.728 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:25:01.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-21T23:25:01.759 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:25:01.760 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-21T23:25:01.830 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:25:01.830 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-21T23:25:01.900 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:25:01.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-21T23:25:01.971 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:25:01.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-21T23:25:02.042 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:25:02.042 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-21T23:25:02.073 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:25:02.073 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-21T23:25:02.149 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:25:02.149 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-21T23:25:02.222 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:25:02.222 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-21T23:25:02.292 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:25:02.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-21T23:25:02.361 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:25:02.361 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-21T23:25:02.380 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:25:02.380 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-21T23:25:02.419 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T23:25:02.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T23:25:02.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T23:25:02.463 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.y] 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCvVe9mozumFxAAnHBGwtGaJBQTJsIcEUqpNQ== 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCyVe9mMdpmMRAAaxr/uVlsBLuo6ycpT7yftg== 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQC2Ve9mHXOuFRAAmvk7cE6MyfMBkgcBoi/9ww== 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQC5Ve9mEWzuJhAA0k2eG3OdoU7d2+xb9173SQ== 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.3] 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQC8Ve9mUEURKRAAPSipSPEJ4QB6CmfHkUMPvw== 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCvVe9mpeVwHRAAFkUcVGK/ArHJIS5zDnO+eA== 2024-09-21T23:25:02.464 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.4] 2024-09-21T23:25:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDAVe9maoofNxAAxlqWVczMj8kiKh/wpBd9vA== 2024-09-21T23:25:02.465 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.5] 2024-09-21T23:25:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDDVe9m0WQ6OhAAnX3L0Kt06MKxt0ZovJghUA== 2024-09-21T23:25:02.465 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.6] 2024-09-21T23:25:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDHVe9m2FQQHhAA21F8iijWLxwqfhJaJkXrcw== 2024-09-21T23:25:02.465 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.7] 2024-09-21T23:25:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDKVe9mmhL1HxAA7+bj9/hFzhm+vE8WnogBmw== 2024-09-21T23:25:02.465 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-21T23:25:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCvVe9mVHsqIBAAMLCxbYYgNgp92z1E+hs5rg== 2024-09-21T23:25:02.465 INFO:teuthology.orchestra.run.smithi031.stdout:[client.1] 2024-09-21T23:25:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCvVe9md+qyIxAAjk/tRahBeCwAAQWanQtcRQ== 2024-09-21T23:25:02.487 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.y] 2024-09-21T23:25:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCvVe9mozumFxAAnHBGwtGaJBQTJsIcEUqpNQ== 2024-09-21T23:25:02.487 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2024-09-21T23:25:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCyVe9mMdpmMRAAaxr/uVlsBLuo6ycpT7yftg== 2024-09-21T23:25:02.487 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC2Ve9mHXOuFRAAmvk7cE6MyfMBkgcBoi/9ww== 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC5Ve9mEWzuJhAA0k2eG3OdoU7d2+xb9173SQ== 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.3] 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC8Ve9mUEURKRAAPSipSPEJ4QB6CmfHkUMPvw== 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCvVe9mpeVwHRAAFkUcVGK/ArHJIS5zDnO+eA== 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.4] 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDAVe9maoofNxAAxlqWVczMj8kiKh/wpBd9vA== 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.5] 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDDVe9m0WQ6OhAAnX3L0Kt06MKxt0ZovJghUA== 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.6] 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDHVe9m2FQQHhAA21F8iijWLxwqfhJaJkXrcw== 2024-09-21T23:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.7] 2024-09-21T23:25:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDKVe9mmhL1HxAA7+bj9/hFzhm+vE8WnogBmw== 2024-09-21T23:25:02.489 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-09-21T23:25:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCvVe9mVHsqIBAAMLCxbYYgNgp92z1E+hs5rg== 2024-09-21T23:25:02.489 INFO:teuthology.orchestra.run.smithi060.stdout:[client.1] 2024-09-21T23:25:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCvVe9md+qyIxAAjk/tRahBeCwAAQWanQtcRQ== 2024-09-21T23:25:02.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T23:25:02.509 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T23:25:02.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.648 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.704 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.760 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.812 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T23:25:02.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T23:25:02.866 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:02.976 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:03.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:03.030 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:25:03.082 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:25:03.084 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:25:03.139 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:25:03.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:25:03.196 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T23:25:03.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-21T23:25:03.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T23:25:03.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-21T23:25:03.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-21T23:25:03.412 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T23:25:03.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-21T23:25:03.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-21T23:25:03.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T23:25:03.666 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-21T23:25:03.692 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:25:03.694 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T23:25:03.748 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-21T23:25:03.748 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-21T23:25:03.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-21T23:25:03.751 INFO:tasks.ceph.mon.a:Started 2024-09-21T23:25:03.751 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-21T23:25:03.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-21T23:25:03.753 INFO:tasks.ceph.mon.c:Started 2024-09-21T23:25:03.753 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-21T23:25:03.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-21T23:25:03.790 INFO:tasks.ceph.mon.b:Started 2024-09-21T23:25:03.791 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-21T23:25:03.791 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-21T23:25:03.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-21T23:25:03.793 INFO:tasks.ceph.mgr.y:Started 2024-09-21T23:25:03.793 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-21T23:25:03.793 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-21T23:25:03.795 INFO:tasks.ceph.mgr.x:Started 2024-09-21T23:25:03.795 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T23:25:03.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd crush tunables default 2024-09-21T23:25:09.157 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-09-21T23:25:09.169 INFO:tasks.ceph:check_enable_crimson: False 2024-09-21T23:25:09.170 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-21T23:25:09.170 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:25:09.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-21T23:25:09.196 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:25:09.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-21T23:25:09.262 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:25:09.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-21T23:25:09.330 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:25:09.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-21T23:25:09.396 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:25:09.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-21T23:25:09.428 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:25:09.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-21T23:25:09.498 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:25:09.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-21T23:25:09.566 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:25:09.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-21T23:25:09.598 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:09.597+0000 7f0147199040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T23:25:09.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af 0 2024-09-21T23:25:09.643 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:09.642+0000 7f0147199040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T23:25:09.818 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:09.817+0000 7f0147199040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T23:25:09.864 INFO:teuthology.orchestra.run.smithi060.stdout:0 2024-09-21T23:25:09.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 88ee06a3-9a6a-41e8-9327-a7a61304c03d 1 2024-09-21T23:25:09.876 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:09.875+0000 7f0147199040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T23:25:09.974 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:09.973+0000 7f38daf23040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T23:25:10.017 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:10.016+0000 7f38daf23040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T23:25:10.055 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:10.053+0000 7f0147199040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T23:25:10.063 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-09-21T23:25:10.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05 2 2024-09-21T23:25:10.161 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:10.161+0000 7f38daf23040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T23:25:10.174 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:10.173+0000 7f0147199040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T23:25:10.214 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:10.214+0000 7f38daf23040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T23:25:10.259 INFO:teuthology.orchestra.run.smithi060.stdout:2 2024-09-21T23:25:10.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 562e8c30-0622-4cb3-b5d1-f15b42a444bc 3 2024-09-21T23:25:10.382 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:10.381+0000 7f38daf23040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T23:25:10.451 INFO:teuthology.orchestra.run.smithi060.stdout:3 2024-09-21T23:25:10.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 7030ff68-4556-4ad5-abc9-be758f1caa85 4 2024-09-21T23:25:10.497 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:10.497+0000 7f38daf23040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T23:25:10.517 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:10.516+0000 7f0147199040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T23:25:10.578 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:10.577+0000 7f0147199040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T23:25:10.646 INFO:teuthology.orchestra.run.smithi060.stdout:4 2024-09-21T23:25:10.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 7169a163-f067-4385-b734-f58a3ba9e89e 5 2024-09-21T23:25:10.739 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:10.738+0000 7f0147199040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T23:25:10.839 INFO:teuthology.orchestra.run.smithi060.stdout:5 2024-09-21T23:25:10.840 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:10.840+0000 7f38daf23040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T23:25:10.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new e511295d-2177-4dcb-a74b-b16b92534df6 6 2024-09-21T23:25:10.900 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:10.900+0000 7f38daf23040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T23:25:11.033 INFO:teuthology.orchestra.run.smithi060.stdout:6 2024-09-21T23:25:11.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 544f8a91-ce82-4855-85eb-ea814eb3d058 7 2024-09-21T23:25:11.058 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:11.058+0000 7f38daf23040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T23:25:11.229 INFO:teuthology.orchestra.run.smithi060.stdout:7 2024-09-21T23:25:11.241 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T23:25:11.241 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-21T23:25:11.243 INFO:tasks.ceph.osd.0:Started 2024-09-21T23:25:11.243 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-21T23:25:11.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-21T23:25:11.246 INFO:tasks.ceph.osd.1:Started 2024-09-21T23:25:11.246 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T23:25:11.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-21T23:25:11.248 INFO:tasks.ceph.osd.2:Started 2024-09-21T23:25:11.248 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-21T23:25:11.248 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-21T23:25:11.250 INFO:tasks.ceph.osd.3:Started 2024-09-21T23:25:11.250 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-21T23:25:11.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-21T23:25:11.253 INFO:tasks.ceph.osd.4:Started 2024-09-21T23:25:11.253 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-21T23:25:11.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-21T23:25:11.255 INFO:tasks.ceph.osd.5:Started 2024-09-21T23:25:11.255 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-21T23:25:11.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-21T23:25:11.257 INFO:tasks.ceph.osd.6:Started 2024-09-21T23:25:11.257 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-21T23:25:11.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-21T23:25:11.260 INFO:tasks.ceph.osd.7:Started 2024-09-21T23:25:11.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T23:25:11.447 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:11.447 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"fsid":"d51cc561-ddb8-41ef-b61c-7c6d698e4dd0","created":"2024-09-21T23:25:09.056809+0000","modified":"2024-09-21T23:25:11.226810+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T23:25:11.226810+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"88ee06a3-9a6a-41e8-9327-a7a61304c03d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"562e8c30-0622-4cb3-b5d1-f15b42a444bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7030ff68-4556-4ad5-abc9-be758f1caa85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7169a163-f067-4385-b734-f58a3ba9e89e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e511295d-2177-4dcb-a74b-b16b92534df6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"544f8a91-ce82-4855-85eb-ea814eb3d058","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:25:11.459 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-21T23:25:11.460 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T23:25:11.770 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:11.769+0000 7f0147199040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T23:25:11.826 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:11.824+0000 7f0147199040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T23:25:12.096 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:12.096+0000 7f38daf23040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T23:25:12.151 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:12.151+0000 7f38daf23040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T23:25:12.212 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:12.211+0000 7f0147199040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T23:25:12.218 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:12.217+0000 7f60e4411640 -1 Falling back to public interface 2024-09-21T23:25:12.265 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T23:25:12.301 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:12.300+0000 7f0147199040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T23:25:12.414 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:12.413+0000 7f0147199040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T23:25:12.427 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-21T23:25:12.428 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":10,"fsid":"d51cc561-ddb8-41ef-b61c-7c6d698e4dd0","created":"2024-09-21T23:25:09.056809+0000","modified":"2024-09-21T23:25:11.226810+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T23:25:11.226810+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"88ee06a3-9a6a-41e8-9327-a7a61304c03d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"562e8c30-0622-4cb3-b5d1-f15b42a444bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7030ff68-4556-4ad5-abc9-be758f1caa85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7169a163-f067-4385-b734-f58a3ba9e89e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e511295d-2177-4dcb-a74b-b16b92534df6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"544f8a91-ce82-4855-85eb-ea814eb3d058","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:25:12.437 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-21T23:25:12.462 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:12.462+0000 7f0147199040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T23:25:12.484 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:12.484+0000 7fb014c3b640 -1 Falling back to public interface 2024-09-21T23:25:12.489 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:12.489+0000 7fb33d3b2640 -1 Falling back to public interface 2024-09-21T23:25:12.539 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:12.539+0000 7f38daf23040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T23:25:12.585 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:12.585+0000 7f0147199040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T23:25:12.628 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:12.627+0000 7f38daf23040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T23:25:12.641 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:12.641+0000 7f0147199040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T23:25:12.688 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:12.688+0000 7f0147199040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T23:25:12.725 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:12.724+0000 7f9fc2e1c640 -1 Falling back to public interface 2024-09-21T23:25:12.726 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:12.725+0000 7f16c6ddb640 -1 Falling back to public interface 2024-09-21T23:25:12.729 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:12.729+0000 7fc470f34640 -1 Falling back to public interface 2024-09-21T23:25:12.740 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:12.740+0000 7f38daf23040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T23:25:12.789 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:12.788+0000 7f38daf23040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T23:25:12.807 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:12.806+0000 7f0147199040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T23:25:12.859 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:12.859+0000 7f0147199040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T23:25:12.907 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:12.907+0000 7f38daf23040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T23:25:12.913 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:12.912+0000 7f0147199040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T23:25:12.960 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:12.960+0000 7f58177cc640 -1 Falling back to public interface 2024-09-21T23:25:12.962 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:12.962+0000 7f38daf23040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T23:25:12.984 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:12.983+0000 7f0147199040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T23:25:13.009 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:13.009+0000 7f38daf23040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T23:25:13.031 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:13.030+0000 7f0147199040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T23:25:13.077 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:13.077+0000 7f0147199040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T23:25:13.127 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:13.127+0000 7f38daf23040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T23:25:13.179 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:13.179+0000 7f38daf23040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T23:25:13.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:13.197+0000 7fc55500e640 -1 Falling back to public interface 2024-09-21T23:25:13.233 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:13.233+0000 7f38daf23040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T23:25:13.304 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:13.303+0000 7f38daf23040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T23:25:13.350 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:13.350+0000 7f38daf23040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T23:25:13.396 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:13.396+0000 7f38daf23040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T23:25:13.764 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:13.764+0000 7f0147199040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T23:25:14.084 INFO:tasks.ceph.mgr.y.smithi031.stderr:2024-09-21T23:25:14.083+0000 7f38daf23040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T23:25:14.780 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:14.779+0000 7f00e7a69640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-21T23:25:14.780 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-21T23:25:14.780+0000 7f00e7a69640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T23:25:15.794 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:15.793+0000 7f60e4411640 -1 osd.3 0 log_to_monitors true 2024-09-21T23:25:16.054 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:16.054+0000 7fb014c3b640 -1 osd.6 0 log_to_monitors true 2024-09-21T23:25:16.316 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:16.315+0000 7fb33d3b2640 -1 osd.4 0 log_to_monitors true 2024-09-21T23:25:16.507 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:16.506+0000 7f9fc2e1c640 -1 osd.7 0 log_to_monitors true 2024-09-21T23:25:16.534 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:16.533+0000 7fc470f34640 -1 osd.0 0 log_to_monitors true 2024-09-21T23:25:16.789 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:16.789+0000 7f16c6ddb640 -1 osd.2 0 log_to_monitors true 2024-09-21T23:25:16.957 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:16.957+0000 7fc55500e640 -1 osd.1 0 log_to_monitors true 2024-09-21T23:25:16.979 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:16.979+0000 7f58177cc640 -1 osd.5 0 log_to_monitors true 2024-09-21T23:25:17.837 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:17.836+0000 7f60e05a6640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T23:25:17.841 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:17.841+0000 7fb010bca640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T23:25:17.848 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:17.848+0000 7f60d7bb2640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:25:17.856 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:17.856+0000 7fb0089ea640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:25:19.070 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:19.070+0000 7fb339341640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T23:25:19.071 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:19.071+0000 7f9fbe7a1640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T23:25:19.081 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:19.081+0000 7f9fb65c1640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:25:19.081 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:19.081+0000 7fb331161640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:25:19.123 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:19.122+0000 7f581375b640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T23:25:19.140 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:19.140+0000 7f580b57b640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:25:19.146 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:19.146+0000 7f16c2d6a640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T23:25:19.163 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:19.163+0000 7f16bab8a640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:25:19.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:19.195+0000 7fc5508fe640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T23:25:19.206 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:19.205+0000 7fc46cec3640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T23:25:19.212 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:19.212+0000 7fc54871e640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:25:19.220 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:19.220+0000 7fc464ce3640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:25:19.243 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T23:25:19.524 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-21T23:25:19.524 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":14,"fsid":"d51cc561-ddb8-41ef-b61c-7c6d698e4dd0","created":"2024-09-21T23:25:09.056809+0000","modified":"2024-09-21T23:25:19.097657+0000","last_up_change":"2024-09-21T23:25:19.097657+0000","last_in_change":"2024-09-21T23:25:11.226810+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"88ee06a3-9a6a-41e8-9327-a7a61304c03d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"562e8c30-0622-4cb3-b5d1-f15b42a444bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6801","nonce":1713632981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6803","nonce":1713632981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6807","nonce":1713632981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6805","nonce":1713632981}]},"public_addr":"172.21.15.31:6801/1713632981","cluster_addr":"172.21.15.31:6803/1713632981","heartbeat_back_addr":"172.21.15.31:6807/1713632981","heartbeat_front_addr":"172.21.15.31:6805/1713632981","state":["exists","up"]},{"osd":4,"uuid":"7030ff68-4556-4ad5-abc9-be758f1caa85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6809","nonce":2660753894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6811","nonce":2660753894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6815","nonce":2660753894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6813","nonce":2660753894}]},"public_addr":"172.21.15.60:6809/2660753894","cluster_addr":"172.21.15.60:6811/2660753894","heartbeat_back_addr":"172.21.15.60:6815/2660753894","heartbeat_front_addr":"172.21.15.60:6813/2660753894","state":["exists","up"]},{"osd":5,"uuid":"7169a163-f067-4385-b734-f58a3ba9e89e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e511295d-2177-4dcb-a74b-b16b92534df6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6801","nonce":3490482162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6803","nonce":3490482162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6807","nonce":3490482162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6805","nonce":3490482162}]},"public_addr":"172.21.15.60:6801/3490482162","cluster_addr":"172.21.15.60:6803/3490482162","heartbeat_back_addr":"172.21.15.60:6807/3490482162","heartbeat_front_addr":"172.21.15.60:6805/3490482162","state":["exists","up"]},{"osd":7,"uuid":"544f8a91-ce82-4855-85eb-ea814eb3d058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6817","nonce":1315554966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6819","nonce":1315554966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6823","nonce":1315554966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6821","nonce":1315554966}]},"public_addr":"172.21.15.60:6817/1315554966","cluster_addr":"172.21.15.60:6819/1315554966","heartbeat_back_addr":"172.21.15.60:6823/1315554966","heartbeat_front_addr":"172.21.15.60:6821/1315554966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:16.758871+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.055855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:25:19.535 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-09-21T23:25:26.342 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T23:25:26.637 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-21T23:25:26.637 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":17,"fsid":"d51cc561-ddb8-41ef-b61c-7c6d698e4dd0","created":"2024-09-21T23:25:09.056809+0000","modified":"2024-09-21T23:25:22.131082+0000","last_up_change":"2024-09-21T23:25:20.126996+0000","last_in_change":"2024-09-21T23:25:11.226810+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:25:19.818212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6817","nonce":2339966024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6819","nonce":2339966024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6823","nonce":2339966024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6821","nonce":2339966024}]},"public_addr":"172.21.15.31:6817/2339966024","cluster_addr":"172.21.15.31:6819/2339966024","heartbeat_back_addr":"172.21.15.31:6823/2339966024","heartbeat_front_addr":"172.21.15.31:6821/2339966024","state":["exists","up"]},{"osd":1,"uuid":"88ee06a3-9a6a-41e8-9327-a7a61304c03d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6825","nonce":2889649822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6827","nonce":2889649822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6831","nonce":2889649822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6829","nonce":2889649822}]},"public_addr":"172.21.15.31:6825/2889649822","cluster_addr":"172.21.15.31:6827/2889649822","heartbeat_back_addr":"172.21.15.31:6831/2889649822","heartbeat_front_addr":"172.21.15.31:6829/2889649822","state":["exists","up"]},{"osd":2,"uuid":"f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6809","nonce":864917651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6811","nonce":864917651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6815","nonce":864917651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6813","nonce":864917651}]},"public_addr":"172.21.15.31:6809/864917651","cluster_addr":"172.21.15.31:6811/864917651","heartbeat_back_addr":"172.21.15.31:6815/864917651","heartbeat_front_addr":"172.21.15.31:6813/864917651","state":["exists","up"]},{"osd":3,"uuid":"562e8c30-0622-4cb3-b5d1-f15b42a444bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6801","nonce":1713632981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6803","nonce":1713632981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6807","nonce":1713632981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6805","nonce":1713632981}]},"public_addr":"172.21.15.31:6801/1713632981","cluster_addr":"172.21.15.31:6803/1713632981","heartbeat_back_addr":"172.21.15.31:6807/1713632981","heartbeat_front_addr":"172.21.15.31:6805/1713632981","state":["exists","up"]},{"osd":4,"uuid":"7030ff68-4556-4ad5-abc9-be758f1caa85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6809","nonce":2660753894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6811","nonce":2660753894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6815","nonce":2660753894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6813","nonce":2660753894}]},"public_addr":"172.21.15.60:6809/2660753894","cluster_addr":"172.21.15.60:6811/2660753894","heartbeat_back_addr":"172.21.15.60:6815/2660753894","heartbeat_front_addr":"172.21.15.60:6813/2660753894","state":["exists","up"]},{"osd":5,"uuid":"7169a163-f067-4385-b734-f58a3ba9e89e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6825","nonce":2320687649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6827","nonce":2320687649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6831","nonce":2320687649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6829","nonce":2320687649}]},"public_addr":"172.21.15.60:6825/2320687649","cluster_addr":"172.21.15.60:6827/2320687649","heartbeat_back_addr":"172.21.15.60:6831/2320687649","heartbeat_front_addr":"172.21.15.60:6829/2320687649","state":["exists","up"]},{"osd":6,"uuid":"e511295d-2177-4dcb-a74b-b16b92534df6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6801","nonce":3490482162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6803","nonce":3490482162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6807","nonce":3490482162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6805","nonce":3490482162}]},"public_addr":"172.21.15.60:6801/3490482162","cluster_addr":"172.21.15.60:6803/3490482162","heartbeat_back_addr":"172.21.15.60:6807/3490482162","heartbeat_front_addr":"172.21.15.60:6805/3490482162","state":["exists","up"]},{"osd":7,"uuid":"544f8a91-ce82-4855-85eb-ea814eb3d058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6817","nonce":1315554966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6819","nonce":1315554966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6823","nonce":1315554966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6821","nonce":1315554966}]},"public_addr":"172.21.15.60:6817/1315554966","cluster_addr":"172.21.15.60:6819/1315554966","heartbeat_back_addr":"172.21.15.60:6823/1315554966","heartbeat_front_addr":"172.21.15.60:6821/1315554966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.514872+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.912129+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.745154+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:16.758871+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.287462+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:18.031636+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.055855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.524609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:25:26.648 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-21T23:25:26.649 INFO:tasks.ceph:Creating RBD pool 2024-09-21T23:25:26.649 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-21T23:25:27.158 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'rbd' created 2024-09-21T23:25:27.171 DEBUG:teuthology.orchestra.run.smithi031:> rbd --cluster ceph pool init rbd 2024-09-21T23:25:30.193 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-21T23:25:30.193 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T23:25:30.193 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T23:25:30.194 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T23:25:30.194 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T23:25:30.548 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:30.559 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2042361600},{"type":"v1","addr":"172.21.15.60:6833","nonce":2042361600}]},"active_addr":"172.21.15.60:6833/2042361600","active_change":"2024-09-21T23:25:13.766950+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4115,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":1672256448}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":125215748}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":3116157232}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":1716961296}]}]} 2024-09-21T23:25:30.563 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T23:25:30.563 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T23:25:30.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T23:25:30.854 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:30.854 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"d51cc561-ddb8-41ef-b61c-7c6d698e4dd0","created":"2024-09-21T23:25:09.056809+0000","modified":"2024-09-21T23:25:30.171098+0000","last_up_change":"2024-09-21T23:25:20.126996+0000","last_in_change":"2024-09-21T23:25:11.226810+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:25:19.818212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T23:25:26.968096+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6817","nonce":2339966024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6819","nonce":2339966024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6823","nonce":2339966024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6821","nonce":2339966024}]},"public_addr":"172.21.15.31:6817/2339966024","cluster_addr":"172.21.15.31:6819/2339966024","heartbeat_back_addr":"172.21.15.31:6823/2339966024","heartbeat_front_addr":"172.21.15.31:6821/2339966024","state":["exists","up"]},{"osd":1,"uuid":"88ee06a3-9a6a-41e8-9327-a7a61304c03d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6825","nonce":2889649822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6827","nonce":2889649822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6831","nonce":2889649822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6829","nonce":2889649822}]},"public_addr":"172.21.15.31:6825/2889649822","cluster_addr":"172.21.15.31:6827/2889649822","heartbeat_back_addr":"172.21.15.31:6831/2889649822","heartbeat_front_addr":"172.21.15.31:6829/2889649822","state":["exists","up"]},{"osd":2,"uuid":"f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6809","nonce":864917651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6811","nonce":864917651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6815","nonce":864917651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6813","nonce":864917651}]},"public_addr":"172.21.15.31:6809/864917651","cluster_addr":"172.21.15.31:6811/864917651","heartbeat_back_addr":"172.21.15.31:6815/864917651","heartbeat_front_addr":"172.21.15.31:6813/864917651","state":["exists","up"]},{"osd":3,"uuid":"562e8c30-0622-4cb3-b5d1-f15b42a444bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6801","nonce":1713632981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6803","nonce":1713632981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6807","nonce":1713632981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6805","nonce":1713632981}]},"public_addr":"172.21.15.31:6801/1713632981","cluster_addr":"172.21.15.31:6803/1713632981","heartbeat_back_addr":"172.21.15.31:6807/1713632981","heartbeat_front_addr":"172.21.15.31:6805/1713632981","state":["exists","up"]},{"osd":4,"uuid":"7030ff68-4556-4ad5-abc9-be758f1caa85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6809","nonce":2660753894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6811","nonce":2660753894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6815","nonce":2660753894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6813","nonce":2660753894}]},"public_addr":"172.21.15.60:6809/2660753894","cluster_addr":"172.21.15.60:6811/2660753894","heartbeat_back_addr":"172.21.15.60:6815/2660753894","heartbeat_front_addr":"172.21.15.60:6813/2660753894","state":["exists","up"]},{"osd":5,"uuid":"7169a163-f067-4385-b734-f58a3ba9e89e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6825","nonce":2320687649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6827","nonce":2320687649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6831","nonce":2320687649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6829","nonce":2320687649}]},"public_addr":"172.21.15.60:6825/2320687649","cluster_addr":"172.21.15.60:6827/2320687649","heartbeat_back_addr":"172.21.15.60:6831/2320687649","heartbeat_front_addr":"172.21.15.60:6829/2320687649","state":["exists","up"]},{"osd":6,"uuid":"e511295d-2177-4dcb-a74b-b16b92534df6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6801","nonce":3490482162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6803","nonce":3490482162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6807","nonce":3490482162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6805","nonce":3490482162}]},"public_addr":"172.21.15.60:6801/3490482162","cluster_addr":"172.21.15.60:6803/3490482162","heartbeat_back_addr":"172.21.15.60:6807/3490482162","heartbeat_front_addr":"172.21.15.60:6805/3490482162","state":["exists","up"]},{"osd":7,"uuid":"544f8a91-ce82-4855-85eb-ea814eb3d058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6817","nonce":1315554966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6819","nonce":1315554966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6823","nonce":1315554966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6821","nonce":1315554966}]},"public_addr":"172.21.15.60:6817/1315554966","cluster_addr":"172.21.15.60:6819/1315554966","heartbeat_back_addr":"172.21.15.60:6823/1315554966","heartbeat_front_addr":"172.21.15.60:6821/1315554966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.514872+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.912129+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.745154+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:16.758871+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.287462+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:18.031636+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.055855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.524609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:25:30.866 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T23:25:30.866 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T23:25:31.157 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:31.157 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"d51cc561-ddb8-41ef-b61c-7c6d698e4dd0","created":"2024-09-21T23:25:09.056809+0000","modified":"2024-09-21T23:25:30.171098+0000","last_up_change":"2024-09-21T23:25:20.126996+0000","last_in_change":"2024-09-21T23:25:11.226810+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:25:19.818212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T23:25:26.968096+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6817","nonce":2339966024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6819","nonce":2339966024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6823","nonce":2339966024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6821","nonce":2339966024}]},"public_addr":"172.21.15.31:6817/2339966024","cluster_addr":"172.21.15.31:6819/2339966024","heartbeat_back_addr":"172.21.15.31:6823/2339966024","heartbeat_front_addr":"172.21.15.31:6821/2339966024","state":["exists","up"]},{"osd":1,"uuid":"88ee06a3-9a6a-41e8-9327-a7a61304c03d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6825","nonce":2889649822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6827","nonce":2889649822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6831","nonce":2889649822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6829","nonce":2889649822}]},"public_addr":"172.21.15.31:6825/2889649822","cluster_addr":"172.21.15.31:6827/2889649822","heartbeat_back_addr":"172.21.15.31:6831/2889649822","heartbeat_front_addr":"172.21.15.31:6829/2889649822","state":["exists","up"]},{"osd":2,"uuid":"f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6809","nonce":864917651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6811","nonce":864917651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6815","nonce":864917651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6813","nonce":864917651}]},"public_addr":"172.21.15.31:6809/864917651","cluster_addr":"172.21.15.31:6811/864917651","heartbeat_back_addr":"172.21.15.31:6815/864917651","heartbeat_front_addr":"172.21.15.31:6813/864917651","state":["exists","up"]},{"osd":3,"uuid":"562e8c30-0622-4cb3-b5d1-f15b42a444bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6801","nonce":1713632981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6803","nonce":1713632981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6807","nonce":1713632981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6805","nonce":1713632981}]},"public_addr":"172.21.15.31:6801/1713632981","cluster_addr":"172.21.15.31:6803/1713632981","heartbeat_back_addr":"172.21.15.31:6807/1713632981","heartbeat_front_addr":"172.21.15.31:6805/1713632981","state":["exists","up"]},{"osd":4,"uuid":"7030ff68-4556-4ad5-abc9-be758f1caa85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6809","nonce":2660753894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6811","nonce":2660753894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6815","nonce":2660753894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6813","nonce":2660753894}]},"public_addr":"172.21.15.60:6809/2660753894","cluster_addr":"172.21.15.60:6811/2660753894","heartbeat_back_addr":"172.21.15.60:6815/2660753894","heartbeat_front_addr":"172.21.15.60:6813/2660753894","state":["exists","up"]},{"osd":5,"uuid":"7169a163-f067-4385-b734-f58a3ba9e89e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6825","nonce":2320687649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6827","nonce":2320687649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6831","nonce":2320687649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6829","nonce":2320687649}]},"public_addr":"172.21.15.60:6825/2320687649","cluster_addr":"172.21.15.60:6827/2320687649","heartbeat_back_addr":"172.21.15.60:6831/2320687649","heartbeat_front_addr":"172.21.15.60:6829/2320687649","state":["exists","up"]},{"osd":6,"uuid":"e511295d-2177-4dcb-a74b-b16b92534df6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6801","nonce":3490482162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6803","nonce":3490482162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6807","nonce":3490482162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6805","nonce":3490482162}]},"public_addr":"172.21.15.60:6801/3490482162","cluster_addr":"172.21.15.60:6803/3490482162","heartbeat_back_addr":"172.21.15.60:6807/3490482162","heartbeat_front_addr":"172.21.15.60:6805/3490482162","state":["exists","up"]},{"osd":7,"uuid":"544f8a91-ce82-4855-85eb-ea814eb3d058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6817","nonce":1315554966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6819","nonce":1315554966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6823","nonce":1315554966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6821","nonce":1315554966}]},"public_addr":"172.21.15.60:6817/1315554966","cluster_addr":"172.21.15.60:6819/1315554966","heartbeat_back_addr":"172.21.15.60:6823/1315554966","heartbeat_front_addr":"172.21.15.60:6821/1315554966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.514872+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.912129+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.745154+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:16.758871+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.287462+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:18.031636+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.055855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.524609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:25:31.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T23:25:31.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T23:25:31.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T23:25:31.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T23:25:31.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-21T23:25:31.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-21T23:25:31.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-21T23:25:31.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-21T23:25:31.415 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-21T23:25:31.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T23:25:31.425 INFO:teuthology.orchestra.run.smithi031.stdout:60129542149 2024-09-21T23:25:31.425 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T23:25:31.427 INFO:teuthology.orchestra.run.smithi031.stdout:55834574853 2024-09-21T23:25:31.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T23:25:31.428 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-21T23:25:31.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T23:25:31.429 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-21T23:25:31.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T23:25:31.430 INFO:teuthology.orchestra.run.smithi031.stdout:60129542150 2024-09-21T23:25:31.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T23:25:31.432 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-21T23:25:31.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T23:25:31.433 INFO:teuthology.orchestra.run.smithi031.stdout:55834574853 2024-09-21T23:25:31.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-21T23:25:31.993 INFO:teuthology.orchestra.run.smithi031.stdout:55834574852 2024-09-21T23:25:31.997 INFO:teuthology.orchestra.run.smithi031.stdout:64424509444 2024-09-21T23:25:32.013 INFO:teuthology.orchestra.run.smithi031.stdout:60129542147 2024-09-21T23:25:32.015 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.3 2024-09-21T23:25:32.018 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.0 2024-09-21T23:25:32.026 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.4 2024-09-21T23:25:32.055 INFO:teuthology.orchestra.run.smithi031.stdout:64424509444 2024-09-21T23:25:32.068 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.2 2024-09-21T23:25:32.071 INFO:teuthology.orchestra.run.smithi031.stdout:64424509445 2024-09-21T23:25:32.082 INFO:teuthology.orchestra.run.smithi031.stdout:60129542148 2024-09-21T23:25:32.085 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.5 2024-09-21T23:25:32.088 INFO:teuthology.orchestra.run.smithi031.stdout:64424509445 2024-09-21T23:25:32.095 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.7 2024-09-21T23:25:32.106 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.1 2024-09-21T23:25:32.117 INFO:teuthology.orchestra.run.smithi031.stdout:55834574852 2024-09-21T23:25:32.129 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.6 2024-09-21T23:25:33.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T23:25:33.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T23:25:33.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T23:25:33.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T23:25:33.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T23:25:33.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T23:25:33.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T23:25:33.130 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-21T23:25:33.565 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-21T23:25:33.571 INFO:teuthology.orchestra.run.smithi031.stdout:55834574853 2024-09-21T23:25:33.587 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.0 2024-09-21T23:25:33.587 DEBUG:teuthology.parallel:result is None 2024-09-21T23:25:33.589 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2024-09-21T23:25:33.589 DEBUG:teuthology.parallel:result is None 2024-09-21T23:25:33.621 INFO:teuthology.orchestra.run.smithi031.stdout:60129542149 2024-09-21T23:25:33.639 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.4 2024-09-21T23:25:33.639 DEBUG:teuthology.parallel:result is None 2024-09-21T23:25:33.641 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-21T23:25:33.655 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.2 2024-09-21T23:25:33.655 DEBUG:teuthology.parallel:result is None 2024-09-21T23:25:33.656 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-21T23:25:33.670 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.1 2024-09-21T23:25:33.670 DEBUG:teuthology.parallel:result is None 2024-09-21T23:25:33.693 INFO:teuthology.orchestra.run.smithi031.stdout:55834574853 2024-09-21T23:25:33.704 INFO:teuthology.orchestra.run.smithi031.stdout:60129542150 2024-09-21T23:25:33.706 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.6 2024-09-21T23:25:33.706 DEBUG:teuthology.parallel:result is None 2024-09-21T23:25:33.714 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-21T23:25:33.717 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.7 2024-09-21T23:25:33.717 DEBUG:teuthology.parallel:result is None 2024-09-21T23:25:33.727 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.5 2024-09-21T23:25:33.727 DEBUG:teuthology.parallel:result is None 2024-09-21T23:25:33.728 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T23:25:33.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:25:34.058 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:34.058 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:25:34.070 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-21T23:25:33.840717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":234656,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754740064,"statfs":{"total":773094113280,"available":772853825536,"internally_reserved":0,"allocated":1794048,"data_stored":1750926,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.681927"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.179918+0000","last_change":"2024-09-21T23:25:30.180125+0000","last_active":"2024-09-21T23:25:30.179918+0000","last_peered":"2024-09-21T23:25:30.179918+0000","last_clean":"2024-09-21T23:25:30.179918+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:25:30.179918+0000","last_undegraded":"2024-09-21T23:25:30.179918+0000","last_fullsized":"2024-09-21T23:25:30.179918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:17.515296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033759699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.189119+0000","last_change":"2024-09-21T23:25:30.189119+0000","last_active":"2024-09-21T23:25:30.189119+0000","last_peered":"2024-09-21T23:25:30.189119+0000","last_clean":"2024-09-21T23:25:30.189119+0000","last_became_active":"2024-09-21T23:25:28.165264+0000","last_became_peered":"2024-09-21T23:25:28.165264+0000","last_unstale":"2024-09-21T23:25:30.189119+0000","last_undegraded":"2024-09-21T23:25:30.189119+0000","last_fullsized":"2024-09-21T23:25:30.189119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:39:53.806934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053117699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263565+0000","last_change":"2024-09-21T23:25:30.263565+0000","last_active":"2024-09-21T23:25:30.263565+0000","last_peered":"2024-09-21T23:25:30.263565+0000","last_clean":"2024-09-21T23:25:30.263565+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:25:30.263565+0000","last_undegraded":"2024-09-21T23:25:30.263565+0000","last_fullsized":"2024-09-21T23:25:30.263565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:46:37.545348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042085499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.189121+0000","last_change":"2024-09-21T23:25:30.189121+0000","last_active":"2024-09-21T23:25:30.189121+0000","last_peered":"2024-09-21T23:25:30.189121+0000","last_clean":"2024-09-21T23:25:30.189121+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:25:30.189121+0000","last_undegraded":"2024-09-21T23:25:30.189121+0000","last_fullsized":"2024-09-21T23:25:30.189121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:00:22.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054857199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.188122+0000","last_change":"2024-09-21T23:25:30.186984+0000","last_active":"2024-09-21T23:25:30.188122+0000","last_peered":"2024-09-21T23:25:30.188122+0000","last_clean":"2024-09-21T23:25:30.188122+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:25:30.188122+0000","last_undegraded":"2024-09-21T23:25:30.188122+0000","last_fullsized":"2024-09-21T23:25:30.188122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:01:38.740017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095847300000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263343+0000","last_change":"2024-09-21T23:25:30.263343+0000","last_active":"2024-09-21T23:25:30.263343+0000","last_peered":"2024-09-21T23:25:30.263343+0000","last_clean":"2024-09-21T23:25:30.263343+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:25:30.263343+0000","last_undegraded":"2024-09-21T23:25:30.263343+0000","last_fullsized":"2024-09-21T23:25:30.263343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:50:16.699815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044148899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263501+0000","last_change":"2024-09-21T23:25:30.263501+0000","last_active":"2024-09-21T23:25:30.263501+0000","last_peered":"2024-09-21T23:25:30.263501+0000","last_clean":"2024-09-21T23:25:30.263501+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:25:30.263501+0000","last_undegraded":"2024-09-21T23:25:30.263501+0000","last_fullsized":"2024-09-21T23:25:30.263501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.155893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036082099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.186477+0000","last_change":"2024-09-21T23:25:30.186477+0000","last_active":"2024-09-21T23:25:30.186477+0000","last_peered":"2024-09-21T23:25:30.186477+0000","last_clean":"2024-09-21T23:25:30.186477+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:25:30.186477+0000","last_undegraded":"2024-09-21T23:25:30.186477+0000","last_fullsized":"2024-09-21T23:25:30.186477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:32:47.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044681799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263047+0000","last_change":"2024-09-21T23:25:21.135362+0000","last_active":"2024-09-21T23:25:30.263047+0000","last_peered":"2024-09-21T23:25:30.263047+0000","last_clean":"2024-09-21T23:25:30.263047+0000","last_became_active":"2024-09-21T23:25:21.135038+0000","last_became_peered":"2024-09-21T23:25:21.135038+0000","last_unstale":"2024-09-21T23:25:30.263047+0000","last_undegraded":"2024-09-21T23:25:30.263047+0000","last_fullsized":"2024-09-21T23:25:30.263047+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:42:58.183351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":661637,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":212992,"data_stored":71288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31400,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":71288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":661637,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:25:34.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:25:34.363 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:34.363 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:25:34.375 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-21T23:25:33.840717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":234656,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754740064,"statfs":{"total":773094113280,"available":772853825536,"internally_reserved":0,"allocated":1794048,"data_stored":1750926,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.681927"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.179918+0000","last_change":"2024-09-21T23:25:30.180125+0000","last_active":"2024-09-21T23:25:30.179918+0000","last_peered":"2024-09-21T23:25:30.179918+0000","last_clean":"2024-09-21T23:25:30.179918+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:25:30.179918+0000","last_undegraded":"2024-09-21T23:25:30.179918+0000","last_fullsized":"2024-09-21T23:25:30.179918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:17.515296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033759699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.189119+0000","last_change":"2024-09-21T23:25:30.189119+0000","last_active":"2024-09-21T23:25:30.189119+0000","last_peered":"2024-09-21T23:25:30.189119+0000","last_clean":"2024-09-21T23:25:30.189119+0000","last_became_active":"2024-09-21T23:25:28.165264+0000","last_became_peered":"2024-09-21T23:25:28.165264+0000","last_unstale":"2024-09-21T23:25:30.189119+0000","last_undegraded":"2024-09-21T23:25:30.189119+0000","last_fullsized":"2024-09-21T23:25:30.189119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:39:53.806934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053117699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263565+0000","last_change":"2024-09-21T23:25:30.263565+0000","last_active":"2024-09-21T23:25:30.263565+0000","last_peered":"2024-09-21T23:25:30.263565+0000","last_clean":"2024-09-21T23:25:30.263565+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:25:30.263565+0000","last_undegraded":"2024-09-21T23:25:30.263565+0000","last_fullsized":"2024-09-21T23:25:30.263565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:46:37.545348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042085499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.189121+0000","last_change":"2024-09-21T23:25:30.189121+0000","last_active":"2024-09-21T23:25:30.189121+0000","last_peered":"2024-09-21T23:25:30.189121+0000","last_clean":"2024-09-21T23:25:30.189121+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:25:30.189121+0000","last_undegraded":"2024-09-21T23:25:30.189121+0000","last_fullsized":"2024-09-21T23:25:30.189121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:00:22.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054857199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.188122+0000","last_change":"2024-09-21T23:25:30.186984+0000","last_active":"2024-09-21T23:25:30.188122+0000","last_peered":"2024-09-21T23:25:30.188122+0000","last_clean":"2024-09-21T23:25:30.188122+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:25:30.188122+0000","last_undegraded":"2024-09-21T23:25:30.188122+0000","last_fullsized":"2024-09-21T23:25:30.188122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:01:38.740017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095847300000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263343+0000","last_change":"2024-09-21T23:25:30.263343+0000","last_active":"2024-09-21T23:25:30.263343+0000","last_peered":"2024-09-21T23:25:30.263343+0000","last_clean":"2024-09-21T23:25:30.263343+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:25:30.263343+0000","last_undegraded":"2024-09-21T23:25:30.263343+0000","last_fullsized":"2024-09-21T23:25:30.263343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:50:16.699815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044148899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263501+0000","last_change":"2024-09-21T23:25:30.263501+0000","last_active":"2024-09-21T23:25:30.263501+0000","last_peered":"2024-09-21T23:25:30.263501+0000","last_clean":"2024-09-21T23:25:30.263501+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:25:30.263501+0000","last_undegraded":"2024-09-21T23:25:30.263501+0000","last_fullsized":"2024-09-21T23:25:30.263501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.155893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036082099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.186477+0000","last_change":"2024-09-21T23:25:30.186477+0000","last_active":"2024-09-21T23:25:30.186477+0000","last_peered":"2024-09-21T23:25:30.186477+0000","last_clean":"2024-09-21T23:25:30.186477+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:25:30.186477+0000","last_undegraded":"2024-09-21T23:25:30.186477+0000","last_fullsized":"2024-09-21T23:25:30.186477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:32:47.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044681799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263047+0000","last_change":"2024-09-21T23:25:21.135362+0000","last_active":"2024-09-21T23:25:30.263047+0000","last_peered":"2024-09-21T23:25:30.263047+0000","last_clean":"2024-09-21T23:25:30.263047+0000","last_became_active":"2024-09-21T23:25:21.135038+0000","last_became_peered":"2024-09-21T23:25:21.135038+0000","last_unstale":"2024-09-21T23:25:30.263047+0000","last_undegraded":"2024-09-21T23:25:30.263047+0000","last_fullsized":"2024-09-21T23:25:30.263047+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:42:58.183351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":661637,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":212992,"data_stored":71288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31400,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":71288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":661637,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:25:34.376 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T23:25:34.376 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T23:25:34.376 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T23:25:34.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T23:25:34.700 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:34.700 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T23:25:34.712 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T23:25:34.712 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-21T23:25:34.728 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T23:25:34.728 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-21T23:25:34.728 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-21T23:25:34.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-21T23:25:35.035 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T23:25:35.035 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T23:25:35.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:25:35.324 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:35.324 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:25:35.336 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-21T23:25:33.840717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":234656,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754740064,"statfs":{"total":773094113280,"available":772853825536,"internally_reserved":0,"allocated":1794048,"data_stored":1750926,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.681927"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.179918+0000","last_change":"2024-09-21T23:25:30.180125+0000","last_active":"2024-09-21T23:25:30.179918+0000","last_peered":"2024-09-21T23:25:30.179918+0000","last_clean":"2024-09-21T23:25:30.179918+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:25:30.179918+0000","last_undegraded":"2024-09-21T23:25:30.179918+0000","last_fullsized":"2024-09-21T23:25:30.179918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:17.515296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033759699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.189119+0000","last_change":"2024-09-21T23:25:30.189119+0000","last_active":"2024-09-21T23:25:30.189119+0000","last_peered":"2024-09-21T23:25:30.189119+0000","last_clean":"2024-09-21T23:25:30.189119+0000","last_became_active":"2024-09-21T23:25:28.165264+0000","last_became_peered":"2024-09-21T23:25:28.165264+0000","last_unstale":"2024-09-21T23:25:30.189119+0000","last_undegraded":"2024-09-21T23:25:30.189119+0000","last_fullsized":"2024-09-21T23:25:30.189119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:39:53.806934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053117699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263565+0000","last_change":"2024-09-21T23:25:30.263565+0000","last_active":"2024-09-21T23:25:30.263565+0000","last_peered":"2024-09-21T23:25:30.263565+0000","last_clean":"2024-09-21T23:25:30.263565+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:25:30.263565+0000","last_undegraded":"2024-09-21T23:25:30.263565+0000","last_fullsized":"2024-09-21T23:25:30.263565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:46:37.545348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042085499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.189121+0000","last_change":"2024-09-21T23:25:30.189121+0000","last_active":"2024-09-21T23:25:30.189121+0000","last_peered":"2024-09-21T23:25:30.189121+0000","last_clean":"2024-09-21T23:25:30.189121+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:25:30.189121+0000","last_undegraded":"2024-09-21T23:25:30.189121+0000","last_fullsized":"2024-09-21T23:25:30.189121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:00:22.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054857199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.188122+0000","last_change":"2024-09-21T23:25:30.186984+0000","last_active":"2024-09-21T23:25:30.188122+0000","last_peered":"2024-09-21T23:25:30.188122+0000","last_clean":"2024-09-21T23:25:30.188122+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:25:30.188122+0000","last_undegraded":"2024-09-21T23:25:30.188122+0000","last_fullsized":"2024-09-21T23:25:30.188122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:01:38.740017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095847300000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263343+0000","last_change":"2024-09-21T23:25:30.263343+0000","last_active":"2024-09-21T23:25:30.263343+0000","last_peered":"2024-09-21T23:25:30.263343+0000","last_clean":"2024-09-21T23:25:30.263343+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:25:30.263343+0000","last_undegraded":"2024-09-21T23:25:30.263343+0000","last_fullsized":"2024-09-21T23:25:30.263343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:50:16.699815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044148899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263501+0000","last_change":"2024-09-21T23:25:30.263501+0000","last_active":"2024-09-21T23:25:30.263501+0000","last_peered":"2024-09-21T23:25:30.263501+0000","last_clean":"2024-09-21T23:25:30.263501+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:25:30.263501+0000","last_undegraded":"2024-09-21T23:25:30.263501+0000","last_fullsized":"2024-09-21T23:25:30.263501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.155893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036082099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.186477+0000","last_change":"2024-09-21T23:25:30.186477+0000","last_active":"2024-09-21T23:25:30.186477+0000","last_peered":"2024-09-21T23:25:30.186477+0000","last_clean":"2024-09-21T23:25:30.186477+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:25:30.186477+0000","last_undegraded":"2024-09-21T23:25:30.186477+0000","last_fullsized":"2024-09-21T23:25:30.186477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:32:47.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044681799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263047+0000","last_change":"2024-09-21T23:25:21.135362+0000","last_active":"2024-09-21T23:25:30.263047+0000","last_peered":"2024-09-21T23:25:30.263047+0000","last_clean":"2024-09-21T23:25:30.263047+0000","last_became_active":"2024-09-21T23:25:21.135038+0000","last_became_peered":"2024-09-21T23:25:21.135038+0000","last_unstale":"2024-09-21T23:25:30.263047+0000","last_undegraded":"2024-09-21T23:25:30.263047+0000","last_fullsized":"2024-09-21T23:25:30.263047+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:42:58.183351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":661637,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":212992,"data_stored":71288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31400,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":71288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":661637,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:25:35.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:25:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:25:35.638 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-21T23:25:33.840717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":234656,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754740064,"statfs":{"total":773094113280,"available":772853825536,"internally_reserved":0,"allocated":1794048,"data_stored":1750926,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.681927"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.179918+0000","last_change":"2024-09-21T23:25:30.180125+0000","last_active":"2024-09-21T23:25:30.179918+0000","last_peered":"2024-09-21T23:25:30.179918+0000","last_clean":"2024-09-21T23:25:30.179918+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:25:30.179918+0000","last_undegraded":"2024-09-21T23:25:30.179918+0000","last_fullsized":"2024-09-21T23:25:30.179918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:17.515296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033759699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.189119+0000","last_change":"2024-09-21T23:25:30.189119+0000","last_active":"2024-09-21T23:25:30.189119+0000","last_peered":"2024-09-21T23:25:30.189119+0000","last_clean":"2024-09-21T23:25:30.189119+0000","last_became_active":"2024-09-21T23:25:28.165264+0000","last_became_peered":"2024-09-21T23:25:28.165264+0000","last_unstale":"2024-09-21T23:25:30.189119+0000","last_undegraded":"2024-09-21T23:25:30.189119+0000","last_fullsized":"2024-09-21T23:25:30.189119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:39:53.806934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053117699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263565+0000","last_change":"2024-09-21T23:25:30.263565+0000","last_active":"2024-09-21T23:25:30.263565+0000","last_peered":"2024-09-21T23:25:30.263565+0000","last_clean":"2024-09-21T23:25:30.263565+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:25:30.263565+0000","last_undegraded":"2024-09-21T23:25:30.263565+0000","last_fullsized":"2024-09-21T23:25:30.263565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:46:37.545348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042085499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.189121+0000","last_change":"2024-09-21T23:25:30.189121+0000","last_active":"2024-09-21T23:25:30.189121+0000","last_peered":"2024-09-21T23:25:30.189121+0000","last_clean":"2024-09-21T23:25:30.189121+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:25:30.189121+0000","last_undegraded":"2024-09-21T23:25:30.189121+0000","last_fullsized":"2024-09-21T23:25:30.189121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:00:22.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054857199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.188122+0000","last_change":"2024-09-21T23:25:30.186984+0000","last_active":"2024-09-21T23:25:30.188122+0000","last_peered":"2024-09-21T23:25:30.188122+0000","last_clean":"2024-09-21T23:25:30.188122+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:25:30.188122+0000","last_undegraded":"2024-09-21T23:25:30.188122+0000","last_fullsized":"2024-09-21T23:25:30.188122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:01:38.740017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095847300000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263343+0000","last_change":"2024-09-21T23:25:30.263343+0000","last_active":"2024-09-21T23:25:30.263343+0000","last_peered":"2024-09-21T23:25:30.263343+0000","last_clean":"2024-09-21T23:25:30.263343+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:25:30.263343+0000","last_undegraded":"2024-09-21T23:25:30.263343+0000","last_fullsized":"2024-09-21T23:25:30.263343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:50:16.699815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044148899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263501+0000","last_change":"2024-09-21T23:25:30.263501+0000","last_active":"2024-09-21T23:25:30.263501+0000","last_peered":"2024-09-21T23:25:30.263501+0000","last_clean":"2024-09-21T23:25:30.263501+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:25:30.263501+0000","last_undegraded":"2024-09-21T23:25:30.263501+0000","last_fullsized":"2024-09-21T23:25:30.263501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.155893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036082099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.186477+0000","last_change":"2024-09-21T23:25:30.186477+0000","last_active":"2024-09-21T23:25:30.186477+0000","last_peered":"2024-09-21T23:25:30.186477+0000","last_clean":"2024-09-21T23:25:30.186477+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:25:30.186477+0000","last_undegraded":"2024-09-21T23:25:30.186477+0000","last_fullsized":"2024-09-21T23:25:30.186477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:32:47.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044681799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T23:25:30.263047+0000","last_change":"2024-09-21T23:25:21.135362+0000","last_active":"2024-09-21T23:25:30.263047+0000","last_peered":"2024-09-21T23:25:30.263047+0000","last_clean":"2024-09-21T23:25:30.263047+0000","last_became_active":"2024-09-21T23:25:21.135038+0000","last_became_peered":"2024-09-21T23:25:21.135038+0000","last_unstale":"2024-09-21T23:25:30.263047+0000","last_undegraded":"2024-09-21T23:25:30.263047+0000","last_fullsized":"2024-09-21T23:25:30.263047+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:42:58.183351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":661637,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":212992,"data_stored":71288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31400,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":71288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":266240,"data_stored":661637,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:25:35.639 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T23:25:35.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-21T23:25:35.931 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 22 2024-09-21T23:25:35.931 INFO:teuthology.orchestra.run.smithi031.stdout:fsid d51cc561-ddb8-41ef-b61c-7c6d698e4dd0 2024-09-21T23:25:35.931 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-09-21T23:25:09.056809+0000 2024-09-21T23:25:35.931 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2024-09-21T23:25:33.838673+0000 2024-09-21T23:25:35.931 INFO:teuthology.orchestra.run.smithi031.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T23:25:35.931 INFO:teuthology.orchestra.run.smithi031.stdout:crush_version 5 2024-09-21T23:25:35.931 INFO:teuthology.orchestra.run.smithi031.stdout:full_ratio 0.95 2024-09-21T23:25:35.931 INFO:teuthology.orchestra.run.smithi031.stdout:backfillfull_ratio 0.9 2024-09-21T23:25:35.931 INFO:teuthology.orchestra.run.smithi031.stdout:nearfull_ratio 0.85 2024-09-21T23:25:35.931 INFO:teuthology.orchestra.run.smithi031.stdout:require_min_compat_client luminous 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:min_compat_client jewel 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:require_osd_release squid 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:stretch_mode_enabled false 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 21 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.99 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:max_osd 8 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6816/2339966024,v1:172.21.15.31:6817/2339966024] [v2:172.21.15.31:6818/2339966024,v1:172.21.15.31:6819/2339966024] exists,up e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6824/2889649822,v1:172.21.15.31:6825/2889649822] [v2:172.21.15.31:6826/2889649822,v1:172.21.15.31:6827/2889649822] exists,up 88ee06a3-9a6a-41e8-9327-a7a61304c03d 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6808/864917651,v1:172.21.15.31:6809/864917651] [v2:172.21.15.31:6810/864917651,v1:172.21.15.31:6811/864917651] exists,up f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6800/1713632981,v1:172.21.15.31:6801/1713632981] [v2:172.21.15.31:6802/1713632981,v1:172.21.15.31:6803/1713632981] exists,up 562e8c30-0622-4cb3-b5d1-f15b42a444bc 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6808/2660753894,v1:172.21.15.60:6809/2660753894] [v2:172.21.15.60:6810/2660753894,v1:172.21.15.60:6811/2660753894] exists,up 7030ff68-4556-4ad5-abc9-be758f1caa85 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6824/2320687649,v1:172.21.15.60:6825/2320687649] [v2:172.21.15.60:6826/2320687649,v1:172.21.15.60:6827/2320687649] exists,up 7169a163-f067-4385-b734-f58a3ba9e89e 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6800/3490482162,v1:172.21.15.60:6801/3490482162] [v2:172.21.15.60:6802/3490482162,v1:172.21.15.60:6803/3490482162] exists,up e511295d-2177-4dcb-a74b-b16b92534df6 2024-09-21T23:25:35.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 up in weight 1 up_from 14 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6816/1315554966,v1:172.21.15.60:6817/1315554966] [v2:172.21.15.60:6818/1315554966,v1:172.21.15.60:6819/1315554966] exists,up 544f8a91-ce82-4855-85eb-ea814eb3d058 2024-09-21T23:25:35.943 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6816/2339966024,v1:172.21.15.31:6817/2339966024] [v2:172.21.15.31:6818/2339966024,v1:172.21.15.31:6819/2339966024] exists,up e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af', 'osd.1 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6824/2889649822,v1:172.21.15.31:6825/2889649822] [v2:172.21.15.31:6826/2889649822,v1:172.21.15.31:6827/2889649822] exists,up 88ee06a3-9a6a-41e8-9327-a7a61304c03d', 'osd.2 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6808/864917651,v1:172.21.15.31:6809/864917651] [v2:172.21.15.31:6810/864917651,v1:172.21.15.31:6811/864917651] exists,up f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05', 'osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6800/1713632981,v1:172.21.15.31:6801/1713632981] [v2:172.21.15.31:6802/1713632981,v1:172.21.15.31:6803/1713632981] exists,up 562e8c30-0622-4cb3-b5d1-f15b42a444bc', 'osd.4 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6808/2660753894,v1:172.21.15.60:6809/2660753894] [v2:172.21.15.60:6810/2660753894,v1:172.21.15.60:6811/2660753894] exists,up 7030ff68-4556-4ad5-abc9-be758f1caa85', 'osd.5 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6824/2320687649,v1:172.21.15.60:6825/2320687649] [v2:172.21.15.60:6826/2320687649,v1:172.21.15.60:6827/2320687649] exists,up 7169a163-f067-4385-b734-f58a3ba9e89e', 'osd.6 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6800/3490482162,v1:172.21.15.60:6801/3490482162] [v2:172.21.15.60:6802/3490482162,v1:172.21.15.60:6803/3490482162] exists,up e511295d-2177-4dcb-a74b-b16b92534df6', 'osd.7 up in weight 1 up_from 14 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6816/1315554966,v1:172.21.15.60:6817/1315554966] [v2:172.21.15.60:6818/1315554966,v1:172.21.15.60:6819/1315554966] exists,up 544f8a91-ce82-4855-85eb-ea814eb3d058'] 2024-09-21T23:25:35.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-21T23:25:36.045 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:36.045 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mon.a", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster": "ceph", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "admin_socket_mode": "", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_client_required": "cephx, none", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_cluster_required": "cephx", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_debug": "true", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_service_required": "cephx", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "auth_supported": "cephx", 2024-09-21T23:25:36.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio": "true", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_aio_reap_max": "16", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_async_discard": "false", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_block_size": "4096", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_aio": "false", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_enable_discard": "false", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_flock_retry": "3", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T23:25:36.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_inject_crash": "0", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_ioring": "false", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_ioring_hipri": "false", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bdev_type": "", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_allocator": "hybrid", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_buffered_io": "true", 2024-09-21T23:25:36.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_replay_recovery": "false", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T23:25:36.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_sync_write": "false", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_allocator": "hybrid", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bdev_label_multi": "true", 2024-09-21T23:25:36.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_create": "true", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_db_create": "false", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_db_path": "", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_db_size": "0", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_path": "", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_size": "107374182400", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_wal_create": "false", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_wal_path": "", 2024-09-21T23:25:36.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs": "true", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_autotune": "true", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T23:25:36.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_size": "0", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_type": "2q", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_clone_cow": "true", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T23:25:36.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_mode": "none", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_freelist": "false", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T23:25:36.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-21T23:25:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fail_eio": "false", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T23:25:36.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T23:25:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_blob_size": "0", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T23:25:36.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_spdk_mem": "512", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T23:25:36.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_tracing": "false", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T23:25:36.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T23:25:36.061 INFO:teuthology.orchestra.run.smithi031.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_cluster_require_version": "2", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_require_signatures": "false", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_require_version": "2", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_service_require_signatures": "false", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_service_require_version": "2", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "cephx_sign_messages": "true", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "chdir": "", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "client_acl_type": "", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "client_asio_thread_count": "2", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "client_cache_mid": "0.750000", 2024-09-21T23:25:36.062 INFO:teuthology.orchestra.run.smithi031.stdout: "client_cache_size": "16384", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_caps_release_delay": "5", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_check_pool_perm": "true", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_debug_force_sync_read": "false", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_debug_getattr_caps": "false", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_debug_inject_features": "", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_die_on_failed_remount": "false", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_dirsize_rbytes": "true", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_force_lazyio": "false", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_fs": "", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T23:25:36.063 INFO:teuthology.orchestra.run.smithi031.stdout: "client_inject_release_failure": "false", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_max_inline_size": "4096", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mds_namespace": "", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_metadata": "", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mount_gid": "-1", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mount_timeout": "300", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mount_uid": "-1", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_mountpoint": "/", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_notify_timeout": "10", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc": "true", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_max_objects": "1000", 2024-09-21T23:25:36.064 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_size": "209715200", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_permissions": "true", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_quota": "true", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_quota_df": "true", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_readahead_max_bytes": "0", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_readahead_max_periods": "4", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_readahead_min": "131072", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_reconnect_stale": "false", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_shutdown_timeout": "30", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_snapdir": ".snap", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_tick_interval": "1", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_trace": "", 2024-09-21T23:25:36.065 INFO:teuthology.orchestra.run.smithi031.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "client_use_faked_inos": "false", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "client_use_random_mds": "false", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_graylog": "false", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_graylog_port": "12201", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_monitors": "default=true", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_syslog": "false", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "clog_to_syslog_level": "info", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "-", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_network": "", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_network_interface": "", 2024-09-21T23:25:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zlib_isal": "false", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zlib_level": "5", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "compressor_zstd_level": "1", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_osd_stat_interval": "0", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_seastar_cpu_cores": "", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "crimson_seastar_num_threads": "0", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_location": "", 2024-09-21T23:25:36.067 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_location_hook": "", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_location_hook_timeout": "10", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "daemonize": "false", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "daos_pool": "tank", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_alienstore": "0/5", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_asok": "1/5", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_asok_assert_abort": "false", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_auth": "20/20", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_bdev": "1/3", 2024-09-21T23:25:36.068 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_bluefs": "1/5", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_bluestore": "1/5", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_buffer": "0/1", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_cephsqlite": "0/5", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_client": "0/5", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_compressor": "1/5", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_context": "0/1", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_crimson_interrupt": "0/5", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_crush": "1/1", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_crypto": "1/5", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_cyanstore": "0/5", 2024-09-21T23:25:36.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_dpdk": "1/5", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_eventtrace": "1/5", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_filer": "0/1", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_filestore": "1/3", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_finisher": "1/1", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_fuse": "1/5", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_javaclient": "1/5", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_journal": "1/3", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_journaler": "0/5", 2024-09-21T23:25:36.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_kstore": "1/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_lockdep": "0/1", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mclock": "1/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds": "1/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_balancer": "1/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_locker": "1/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_log": "1/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_migrator": "1/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mds_quiesce": "3/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_memstore": "1/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mgr": "2/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mgrc": "1/5", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_mon": "20/20", 2024-09-21T23:25:36.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_monc": "0/10", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_ms": "1/1", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_none": "0/5", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_objclass": "0/5", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_objectcacher": "0/5", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_objecter": "0/1", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_optracker": "0/5", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_osd": "1/5", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_paxos": "20/20", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_perfcounter": "1/5", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_prioritycache": "1/5", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rados": "0/5", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd": "0/5", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T23:25:36.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rbd_replay": "0/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_refs": "0/0", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_reserver": "1/1", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw": "1/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_access": "1/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_flight": "1/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rgw_sync": "1/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_rocksdb": "4/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore": "0/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_backref": "0/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_cache": "0/5", 2024-09-21T23:25:36.073 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_device": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_epm": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_journal": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_lba": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_odata": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_omap": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_onode": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_t": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_seastore_tm": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_striper": "0/1", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_test": "0/5", 2024-09-21T23:25:36.074 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_throttle": "1/1", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_timer": "0/1", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_tp": "0/5", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "debug_trace": "1/5", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "device_failure_prediction_mode": "none", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_graylog": "false", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_journald": "false", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_stderr": "true", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "err_to_syslog": "false", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "event_tracing": "false", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T23:25:36.075 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_http_port": "9926", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_prio_limit": "5", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_sort_metrics": "true", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "exporter_stats_period": "5", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "fake_statfs_for_testing": "0", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "fatal_signal_handlers": "true", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "filer_max_purge_ops": "10", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "filer_max_truncate_ops": "128", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_blackhole": "false", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_btrfs_snap": "true", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_caller_concurrency": "10", 2024-09-21T23:25:36.076 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_debug_omap_check": "false", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_debug_verify_split": "false", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_dump_file": "", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fadvise": "true", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fail_eio": "true", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fd_cache_size": "128", 2024-09-21T23:25:36.077 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fiemap": "false", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_inject_stall": "0", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_journal_parallel": "false", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_journal_trailing": "false", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_journal_writeahead": "false", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_kill_at": "0", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T23:25:36.078 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_merge_threshold": "-10", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_odsync_write": "false", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_omap_backend_path": "", 2024-09-21T23:25:36.079 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_op_threads": "2", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_punch_hole": "false", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T23:25:36.080 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_queue_max_ops": "50", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_seek_data_hole": "false", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_sloppy_crc": "false", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_splice": "false", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_split_multiple": "2", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_split_rand_factor": "20", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_update_to": "1000", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T23:25:36.081 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_xfs_extsize": "false", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "filestore_zfs_snap": "false", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "fio_dir": "/tmp/fio", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "d51cc561-ddb8-41ef-b61c-7c6d698e4dd0", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_allow_other": "true", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T23:25:36.082 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_big_writes": "true", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_debug": "false", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_default_permissions": "false", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_disable_pagecache": "false", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_max_write": "0", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_multithreaded": "true", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_require_active_mds": "true", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_set_user_groups": "true", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_splice_move": "true", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_splice_read": "true", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_splice_write": "true", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T23:25:36.083 INFO:teuthology.orchestra.run.smithi031.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "gss_target_name": "ceph", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_file": "", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_inject_failure": "0", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_interval": "5", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "host": "smithi031", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T23:25:36.084 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "inject_early_sigterm": "false", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "jaeger_agent_port": "6799", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "jaeger_tracing_enable": "false", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_aio": "true", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_align_min_size": "65536", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_block_align": "true", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_block_size": "4096", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_dio": "true", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_discard": "false", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_force_aio": "false", 2024-09-21T23:25:36.085 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_ignore_corruption": "false", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_max_write_entries": "100", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_replay_from": "0", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_write_header_frequency": "0", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journal_zero_on_create": "false", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journaler_prefetch_periods": "10", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journaler_prezero_periods": "5", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "journaler_write_head_interval": "15", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "keyfile": "", 2024-09-21T23:25:36.086 INFO:teuthology.orchestra.run.smithi031.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_backend": "rocksdb", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_max_bytes": "67108864", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_max_ops": "512", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_onode_map_size": "1024", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "kstore_sync_transaction": "false", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "librados_thread_count": "2", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "lockdep": "false", 2024-09-21T23:25:36.087 INFO:teuthology.orchestra.run.smithi031.stdout: "lockdep_force_backtrace": "false", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_coarse_timestamps": "true", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_flush_on_exit": "false", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_graylog_port": "12201", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_max_new": "1000", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_max_recent": "10000", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_stderr_prefix": "", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": "true", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_graylog": "false", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_journald": "false", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_stderr": "false", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_syslog": "false", 2024-09-21T23:25:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_action_on_write_error": "1", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_alternate_name_max": "8192", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_asio_thread_count": "2", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_export_pin": "true", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_interval": "10", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_max": "-1", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_max_until": "-1", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_merge_size": "50", 2024-09-21T23:25:36.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_mode": "0", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_overload_epochs": "2", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_bits": "3", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_size": "10000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T23:25:36.090 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_mid": "0.700000", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_delay": "0", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_trim_interval": "1", 2024-09-21T23:25:36.091 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_auth_pins": "false", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_frag": "false", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_scatterstat": "false", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_debug_subtrees": "false", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_default_dir_hash": "2", 2024-09-21T23:25:36.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_defer_session_stale": "true", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_max_entries": "0", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dir_prefetch": "true", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_early_reply": "true", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_enable_op_tracker": "true", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_enforce_unique_name": "true", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T23:25:36.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_health_dummy": "false", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T23:25:36.094 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_join_fs": "", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_journal_format": "1", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_create_at": "0", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_export_at": "0", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_import_at": "0", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_journal_at": "0", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_link_at": "0", 2024-09-21T23:25:36.095 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_openc_at": "0", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_rename_at": "0", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_kill_shutdown_at": "0", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_event_large_threshold": "524288", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_max_events": "-1", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_max_segments": "128", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_pause": "false", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_segment_size": "0", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_trim_threshold": "128", 2024-09-21T23:25:36.096 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_completed_requests": "100000", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_export_size": "20971520", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_file_recover": "32", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_purge_files": "64", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_purge_ops": "8192", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_metrics_update_interval": "2", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_min_caps_per_client": "100", 2024-09-21T23:25:36.097 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_numa_node": "-1", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_duration": "600", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_size": "20", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_op_log_threshold": "5", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_ping_grace": "15", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_ping_interval": "5", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T23:25:36.098 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_max_caps": "30000", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_replay_interval": "1.000000", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_root_ino_gid": "0", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_root_ino_uid": "0", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T23:25:36.099 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_shutdown_check": "0", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_skip_ino": "0", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_snap_min_uid": "0", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_snap_rstat": "false", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_symlink_recovery": "true", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T23:25:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_thrash_exports": "0", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_thrash_fragments": "0", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_tick_interval": "5.000000", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_valgrind_exit": "false", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_verify_backtrace": "1", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_verify_scatter": "false", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_wipe_sessions": "false", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool_debug": "false", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_page_set": "false", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "memstore_page_size": "65536", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_client_bytes": "134217728", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_client_messages": "512", 2024-09-21T23:25:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_disabled_modules": "", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_enable_op_tracker": "true", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mds_messages": "128", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_mon_messages": "128", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-21T23:25:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_history_duration": "600", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_history_size": "20", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_op_log_threshold": "5", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_osd_messages": "8192", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_pool": "true", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_standby_modules": "true", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_stats_period": "5", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_stats_threshold": "5", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_tick_period": "2", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T23:25:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_allow_pool_delete": "true", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_allow_pool_size_one": "true", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_bytes": "104857600", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_directed_command_retry": "5", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T23:25:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_client_target_rank": "-1", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_file_level": "debug", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_journald": "false", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-21T23:25:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_compact_on_start": "false", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_compact_on_trim": "true", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_cpu_threads": "4", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data_avail_crit": "5", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data_avail_warn": "5", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T23:25:36.106 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_dump_json": "false", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_extra_checks": "false", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_no_require_squid": "false", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_down_mkfs_grace": "120", 2024-09-21T23:25:36.107 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_down_uptime_grace": "60", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_election_default_strategy": "1", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_election_timeout": "5.000000", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_enable_op_tracker": "true", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_fake_pool_delete": "false", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_force_quorum_join": "false", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_detail_to_clog": "false", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_log_update_period": "5", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_max_detail": "50", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_to_clog": "true", 2024-09-21T23:25:36.108 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_host": "172.21.15.31,[v2:172.21.15.31:3301,v1:172.21.15.31:6790],172.21.15.60", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_host_override": "", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_initial_members": "", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_lease": "5.000000", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_log_full_interval": "50", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_log_max": "10000", 2024-09-21T23:25:36.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_log_max_summary": "50", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_log_epochs": "500", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_osd": "10000", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mds_skip_sanity": "false", 2024-09-21T23:25:36.110 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_memory_autotune": "true", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_memory_target": "2147483648", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_digest_period": "5", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_min_osdmap_epochs": "500", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_complaint_time": "30", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_duration": "600", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_size": "20", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T23:25:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_op_log_threshold": "5", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_cache_size": "500", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T23:25:36.112 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T23:25:36.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_report_timeout": "900", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T23:25:36.114 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_max_osds": "4", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_interval": "86400", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_max_keys": "100", 2024-09-21T23:25:36.115 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_scrub_timeout": "300", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_session_timeout": "300", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_smart_report_timeout": "5", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_pool_size": "4", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_debug": "false", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T23:25:36.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_tick_interval": "5", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T23:25:36.117 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_older_version": "true", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "monmap": "", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T23:25:36.118 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "motr_tracing_enabled": "false", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_op_threads": "3", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_cm": "false", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_device_name": "", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T23:25:36.119 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_sl": "3", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_rdma_type": "ib", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_async_reap_threshold": "5", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_before_connect": "false", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_ipv4": "true", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_ipv6": "false", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_msgr1": "true", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_msgr2": "true", 2024-09-21T23:25:36.120 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_port_max": "7568", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_port_min": "6800", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_retry_count": "3", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_bind_retry_delay": "5", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_client": "false", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_mds": "false", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_mgr": "false", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_mon": "false", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_blackhole_osd": "false", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_client_mode": "crc secure", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_cluster_type": "", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_compress_secure": "false", 2024-09-21T23:25:36.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_crc_data": "true", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_crc_header": "true", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_bug": "true", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_old_message": "true", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T23:25:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_hugepages": "", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_lro": "true", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_pmd": "", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_port_id": "0", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_dump_on_send": "false", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T23:25:36.123 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_delay_type": "", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_network_congestion": "0", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_inject_socket_failures": "5000", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_max_accept_failures": "4", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_max_backoff": "15.000000", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_osd_compress_mode": "none", 2024-09-21T23:25:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_pq_min_cost": "65536", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_public_type": "", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_service_mode": "crc secure", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_nodelay": "true", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "ms_type": "async+posix", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "no_config_file": "false", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T23:25:36.125 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_inflight_ops": "1024", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter_timeout": "10.000000", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "objectstore_blackhole": "false", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "openssl_engine_opts": "", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_max_ops": "4", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T23:25:36.126 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_agent_slop": "0.020000", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_asio_thread_count": "2", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backfill_scan_max": "512", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backfill_scan_min": "64", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backoff_on_peering": "false", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T23:25:36.127 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_beacon_report_interval": "300", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_duration": "30", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_blkin_trace_all": "false", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T23:25:36.128 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_class_update_on_start": "true", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_message_cap": "256", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_op_priority": "63", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_client_watch_timeout": "30", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_command_max_records": "256", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_compact_on_start": "false", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_update_on_start": "true", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T23:25:36.129 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_op_order": "false", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T23:25:36.130 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_shutdown": "false", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_trim_objects": "false", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T23:25:36.131 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_default_notify_timeout": "30", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_enable_op_tracker": "true", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T23:25:36.132 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_info": "true", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_shutdown": "true", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_function_tracing": "false", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_grace": "20", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_interval": "6", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T23:25:36.133 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_stale": "600", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_journal_size": "5120", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_kill_backfill_at": "0", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T23:25:36.134 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_cache_size": "50", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_dedup": "true", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_message_max": "40", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_attr_name_len": "100", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_attr_size": "0", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_backfills": "1", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_markdown_count": "5", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_markdown_period": "600", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_object_name_len": "2048", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_object_size": "134217728", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T23:25:36.135 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_pgls": "1024", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_push_cost": "8388608", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_push_objects": "10", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_scrubs": "3", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_max_write_size": "90", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T23:25:36.136 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_profile": "balanced", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T23:25:36.137 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_base": "805306368", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_target": "4294967296", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_target_autotune": "false", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_min_recovery_priority": "0", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T23:25:36.138 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_report_interval": "5", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_num_cache_shards": "32", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_numa_node": "-1", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objecter_finishers": "1", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objectstore": "bluestore", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objectstore_fuse": "false", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_objectstore_tracing": "false", 2024-09-21T23:25:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_duration": "600", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_size": "20", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_log_threshold": "5", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_shards": "0", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T23:25:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_op_thread_timeout": "15", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_open_classes_on_start": "true", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_os_flags": "0", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_peering_op_priority": "255", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_delete_priority": "5", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T23:25:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_crimson": "false", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T23:25:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_flags": "0", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_min_size": "0", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_read_ratio": "70", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_size": "2", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_default_type": "replicated", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T23:25:36.143 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_cost": "20971520", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_active": "0", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_op_priority": "3", 2024-09-21T23:25:36.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_priority": "5", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_repair_during_recovery": "false", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_requested_scrub_priority": "120", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T23:25:36.145 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_chunk_max": "25", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_cost": "52428800", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_end_hour": "0", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T23:25:36.146 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_priority": "5", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_skip_data_digest": "false", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_smart_report_timeout": "5", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_priority": "5", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T23:25:36.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_target_transaction_size": "30", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T23:25:36.148 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_tracing": "false", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_use_stale_snap": "false", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_kill_at": "0", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_max_join_drift": "10", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_min": "500", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_min_wait": "0.050000", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_service_trim_max": "500", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_service_trim_min": "250", 2024-09-21T23:25:36.149 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_stash_full_interval": "25", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_trim_max": "500", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "paxos_trim_min": "250", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "perf": "true", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "-", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrv": "", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "public_bind_addr": "-", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "public_network": "", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "public_network_interface": "", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "qat_compressor_busy_polling": "false", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "qat_compressor_enabled": "false", 2024-09-21T23:25:36.150 INFO:teuthology.orchestra.run.smithi031.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rados_mon_op_timeout": "0", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rados_osd_op_timeout": "0", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rados_tracing": "false", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_atime_update_interval": "60", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache": "true", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T23:25:36.151 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_policy": "writearound", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_size": "33554432", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_compression_hint": "none", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_clone_format": "auto", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_data_pool": "", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_features": "61", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_format": "2", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_map_options": "", 2024-09-21T23:25:36.152 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_order": "22", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_pool": "rbd", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_stripe_count": "0", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_io_scheduler": "simple", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T23:25:36.153 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_order": "24", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_pool": "", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_journal_splay_width": "4", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T23:25:36.154 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T23:25:36.155 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T23:25:36.156 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_op_threads": "1", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_plugins": "", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T23:25:36.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T23:25:36.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_tracing": "false", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_validate_names": "true", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_validate_pool": "true", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "restapi_base_url": "", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "restapi_log_level": "", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-21T23:25:36.159 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_admin_entry": "admin", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_backend_store": "rados", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_barbican_url": "", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_beast_enable_async": "true", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_counters_cache": "false", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T23:25:36.160 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cache_enabled": "true", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cache_lru_size": "25000", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_config_store": "rados", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_content_length_compat": "false", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_cross_domain_policy": "", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T23:25:36.161 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T23:25:36.162 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T23:25:36.163 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T23:25:36.164 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_d4n_port": "6379", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_log_window": "30", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T23:25:36.165 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T23:25:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dns_name": "", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dns_s3website_name": "", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_ops_log": "false", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T23:25:36.167 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_static_website": "false", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enable_usage_log": "false", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_expose_bucket": "false", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_extended_http_attrs": "", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_filter": "none", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_objs": "32", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T23:25:36.168 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_init_timeout": "300", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_kafka_connection_idle": "300", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T23:25:36.169 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_password": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_project": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_token": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_admin_user": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_api_version": "2", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T23:25:36.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_url": "", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_objs": "32", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_worker": "3", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T23:25:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ldap_uri": "", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_http_headers": "", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T23:25:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_attr_size": "0", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_listing_results": "1000", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_notify_retries": "10", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T23:25:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T23:25:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_num_control_oids": "8", 2024-09-21T23:25:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_numa_node": "-1", 2024-09-21T23:25:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T23:25:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T23:25:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T23:25:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T23:25:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T23:25:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T23:25:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T23:25:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T23:25:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_op_tracing": "false", 2024-09-21T23:25:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_opa_token": "", 2024-09-21T23:25:36.177 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_opa_url": "", 2024-09-21T23:25:36.177 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T23:25:36.177 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T23:25:36.177 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_rados": "false", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_lanes": "3", 2024-09-21T23:25:36.178 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_cache_partitions": "3", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_print_continue": "true", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rados_tracing": "false", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_realm": "", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_realm_id": "", 2024-09-21T23:25:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_region": "", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_request_uri": "", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T23:25:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_resolve_cname": "false", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_run_sync_thread": "true", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T23:25:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_script_uri": "", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_service_provider_name": "", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_client_id": "", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_client_secret": "", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_entry": "sts", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_key": "sts", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_auth_url": "", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_custom_header": "", 2024-09-21T23:25:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_need_stats": "true", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_tenant_name": "", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_url": "", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_lease_period": "120", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T23:25:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_thread_pool_size": "512", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_comment": "", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_createby": "", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_encoding": "", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_flag": "false", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-21T23:25:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_origin": "", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_torrent_tracker": "", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_max_shards": "32", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_use_opa_authz": "false", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_counters_cache": "false", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T23:25:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_user_unique_email": "true", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_verify_ssl": "true", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zone": "", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zone_id": "", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zonegroup": "", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zonegroup_id": "", 2024-09-21T23:25:36.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_block_size": "4096", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T23:25:36.187 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_partition_filters": "false", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_perf": "false", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "run_dir": "/var/run/ceph", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_block_create": "true", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T23:25:36.188 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_device_size": "53687091200", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_full_integrity_check": "false", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_main_device_type": "SSD", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T23:25:36.189 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout: "seastore_segment_size": "67108864", 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout: "set_keepcaps": "false", 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout: "setgroup": "", 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout: "setuser": "", 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout: "setuser_match_path": "", 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout: "thp": "false", 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout: "threadpool_default_timeout": "60", 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout: "throttler_perf_counter": "true" 2024-09-21T23:25:36.190 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:36.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T23:25:36.307 INFO:teuthology.orchestra.run.smithi031.stdout:mon.a: {} 2024-09-21T23:25:36.307 INFO:teuthology.orchestra.run.smithi031.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T23:25:36.317 INFO:teuthology.orchestra.run.smithi031.stdout:mon.b: {} 2024-09-21T23:25:36.317 INFO:teuthology.orchestra.run.smithi031.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T23:25:36.326 INFO:teuthology.orchestra.run.smithi031.stdout:mon.c: {} 2024-09-21T23:25:36.326 INFO:teuthology.orchestra.run.smithi031.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T23:25:36.336 DEBUG:teuthology.orchestra.run.smithi060:> type ceph-objectstore-tool 2024-09-21T23:25:36.355 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T23:25:36.356 DEBUG:teuthology.orchestra.run.smithi031:> type ceph-objectstore-tool 2024-09-21T23:25:36.371 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T23:25:36.371 INFO:teuthology.run_tasks:Running task workunit... 2024-09-21T23:25:36.384 INFO:tasks.workunit:Pulling workunits from ref 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:25:36.384 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T23:25:36.384 DEBUG:teuthology.orchestra.run.smithi031:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T23:25:36.385 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T23:25:36.385 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:25:36.385 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:25:36.385 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T23:25:36.385 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T23:25:36.385 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T23:25:36.386 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T23:25:36.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:25:36.386 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T23:25:36.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:25:36.386 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T23:25:36.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T23:25:36.397 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:36.397+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:25:36.429 INFO:teuthology.orchestra.run.smithi031.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T23:25:36.429 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T23:25:36.487 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:36.487+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:36.491 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T23:25:36.491 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T23:25:36.531 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:36.532 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:36.532 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:36.532 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:36.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:25:36.550 INFO:tasks.workunit:timeout=3h 2024-09-21T23:25:36.551 INFO:tasks.workunit:cleanup=True 2024-09-21T23:25:36.551 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:25:36.553 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:25:36.553 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:25:36.575 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T23:25:36.577 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:25:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:25:36.588 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:36.588+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:36.596 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:25:36.596 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:25:36.613 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:25:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:25:36.630 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:25:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:25:36.648 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:25:36.648 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:25:36.666 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:25:36.666 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:25:36.685 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:25:36.685 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:25:36.689 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:36.689+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:36.692 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:36.692 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:36.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:25:36.735 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:36.735 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:36.735 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 22, 2024-09-21T23:25:36.735 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "d51cc561-ddb8-41ef-b61c-7c6d698e4dd0", 2024-09-21T23:25:36.735 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-09-21T23:25:09.056809+0000", 2024-09-21T23:25:36.735 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-09-21T23:25:33.838673+0000", 2024-09-21T23:25:36.735 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-09-21T23:25:20.126996+0000", 2024-09-21T23:25:36.735 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-09-21T23:25:11.226810+0000", 2024-09-21T23:25:36.735 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T23:25:36.735 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5799936, 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 5, 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 2, 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 8, 2024-09-21T23:25:36.736 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "jewel", 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:19.818212+0000", 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:25:36.737 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 1, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 1, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 1, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 1, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 1, 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:25:36.738 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "17", 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:25:36.739 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:25:36.740 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:25:36.741 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 8, 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 8, 2024-09-21T23:25:36.742 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.25, 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 1, 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 1 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 2, 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "rbd", 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:26.968096+0000", 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 8193, 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:25:36.743 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 8, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 8, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 8, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 8, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 8, 2024-09-21T23:25:36.744 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "21", 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:25:36.745 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:25:36.746 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:25:36.746 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 2, 2024-09-21T23:25:36.746 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 21, 2024-09-21T23:25:36.746 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:25:36.746 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:25:36.746 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:25:36.746 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:25:36.746 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:25:36.746 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:25:36.747 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:25:36.747 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:25:36.747 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:25:36.747 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:25:36.748 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd": {} 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 1.9900000095367432, 2024-09-21T23:25:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 1.9900000095367432, 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 1.75, 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 1.75, 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 1, 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 1 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af", 2024-09-21T23:25:36.750 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.751 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:36.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:36.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6817/2339966024", 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6819/2339966024", 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6823/2339966024", 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6821/2339966024", 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:36.755 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "88ee06a3-9a6a-41e8-9327-a7a61304c03d", 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:25:36.756 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:36.757 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:36.757 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:36.757 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.757 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.757 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-09-21T23:25:36.757 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:36.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.757 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.757 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-09-21T23:25:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-09-21T23:25:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6825/2889649822", 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6827/2889649822", 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6831/2889649822", 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6829/2889649822", 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05", 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-09-21T23:25:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6809/864917651", 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6811/864917651", 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6815/864917651", 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6813/864917651", 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:36.767 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "562e8c30-0622-4cb3-b5d1-f15b42a444bc", 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:36.768 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.770 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6801/1713632981", 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6803/1713632981", 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6807/1713632981", 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6805/1713632981", 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7030ff68-4556-4ad5-abc9-be758f1caa85", 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:36.773 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.774 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6808", 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6809", 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6810", 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6811", 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6814", 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6815", 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.777 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6812", 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6813", 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6809/2660753894", 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6811/2660753894", 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6815/2660753894", 2024-09-21T23:25:36.778 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6813/2660753894", 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7169a163-f067-4385-b734-f58a3ba9e89e", 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:36.779 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6824", 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.780 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6825", 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6826", 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.781 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6827", 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6830", 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.782 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6831", 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6828", 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.783 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6829", 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6825/2320687649", 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6827/2320687649", 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6831/2320687649", 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6829/2320687649", 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.784 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e511295d-2177-4dcb-a74b-b16b92534df6", 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6800", 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6801", 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6802", 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6803", 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:36.787 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.789 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.789 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6806", 2024-09-21T23:25:36.790 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:36.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.790 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.791 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6807", 2024-09-21T23:25:36.791 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:36.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.791 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.791 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:36.792 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.792 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.792 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6804", 2024-09-21T23:25:36.792 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6805", 2024-09-21T23:25:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6801/3490482162", 2024-09-21T23:25:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6803/3490482162", 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3490482162", 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6805/3490482162", 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "544f8a91-ce82-4855-85eb-ea814eb3d058", 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6816", 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6817", 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6818", 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6819", 2024-09-21T23:25:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6822", 2024-09-21T23:25:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6823", 2024-09-21T23:25:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.799 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:36.799 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:36.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:36.799 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6820", 2024-09-21T23:25:36.799 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:36.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6821", 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6817/1315554966", 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6819/1315554966", 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6823/1315554966", 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6821/1315554966", 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.800 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.514872+0000", 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.801 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.912129+0000", 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:36.802 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.745154+0000", 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:36.803 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:16.758871+0000", 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.287462+0000", 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:25:36.804 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:18.031636+0000", 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.055855+0000", 2024-09-21T23:25:36.805 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.524609+0000", 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [], 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [], 2024-09-21T23:25:36.806 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [], 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": {}, 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-09-21T23:25:36.807 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [ 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 2, 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: "snaps": [ 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: "begin": 2, 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: "length": 1 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-09-21T23:25:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-09-21T23:25:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-09-21T23:25:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-09-21T23:25:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-09-21T23:25:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:36.809 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:36.811 INFO:tasks.thrashosds.thrasher:j is {'epoch': 22, 'fsid': 'd51cc561-ddb8-41ef-b61c-7c6d698e4dd0', 'created': '2024-09-21T23:25:09.056809+0000', 'modified': '2024-09-21T23:25:33.838673+0000', 'last_up_change': '2024-09-21T23:25:20.126996+0000', 'last_in_change': '2024-09-21T23:25:11.226810+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T23:25:19.818212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T23:25:26.968096+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 2339966024}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 2339966024}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 2339966024}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 2339966024}]}, 'public_addr': '172.21.15.31:6817/2339966024', 'cluster_addr': '172.21.15.31:6819/2339966024', 'heartbeat_back_addr': '172.21.15.31:6823/2339966024', 'heartbeat_front_addr': '172.21.15.31:6821/2339966024', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '88ee06a3-9a6a-41e8-9327-a7a61304c03d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 2889649822}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2889649822}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2889649822}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2889649822}]}, 'public_addr': '172.21.15.31:6825/2889649822', 'cluster_addr': '172.21.15.31:6827/2889649822', 'heartbeat_back_addr': '172.21.15.31:6831/2889649822', 'heartbeat_front_addr': '172.21.15.31:6829/2889649822', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 864917651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 864917651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 864917651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 864917651}]}, 'public_addr': '172.21.15.31:6809/864917651', 'cluster_addr': '172.21.15.31:6811/864917651', 'heartbeat_back_addr': '172.21.15.31:6815/864917651', 'heartbeat_front_addr': '172.21.15.31:6813/864917651', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '562e8c30-0622-4cb3-b5d1-f15b42a444bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6800', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6801', 'nonce': 1713632981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1713632981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1713632981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1713632981}]}, 'public_addr': '172.21.15.31:6801/1713632981', 'cluster_addr': '172.21.15.31:6803/1713632981', 'heartbeat_back_addr': '172.21.15.31:6807/1713632981', 'heartbeat_front_addr': '172.21.15.31:6805/1713632981', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7030ff68-4556-4ad5-abc9-be758f1caa85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6809', 'nonce': 2660753894}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6811', 'nonce': 2660753894}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6815', 'nonce': 2660753894}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6813', 'nonce': 2660753894}]}, 'public_addr': '172.21.15.60:6809/2660753894', 'cluster_addr': '172.21.15.60:6811/2660753894', 'heartbeat_back_addr': '172.21.15.60:6815/2660753894', 'heartbeat_front_addr': '172.21.15.60:6813/2660753894', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7169a163-f067-4385-b734-f58a3ba9e89e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6824', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6825', 'nonce': 2320687649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6826', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6827', 'nonce': 2320687649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6830', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6831', 'nonce': 2320687649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6828', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6829', 'nonce': 2320687649}]}, 'public_addr': '172.21.15.60:6825/2320687649', 'cluster_addr': '172.21.15.60:6827/2320687649', 'heartbeat_back_addr': '172.21.15.60:6831/2320687649', 'heartbeat_front_addr': '172.21.15.60:6829/2320687649', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e511295d-2177-4dcb-a74b-b16b92534df6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6801', 'nonce': 3490482162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6803', 'nonce': 3490482162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6807', 'nonce': 3490482162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6805', 'nonce': 3490482162}]}, 'public_addr': '172.21.15.60:6801/3490482162', 'cluster_addr': '172.21.15.60:6803/3490482162', 'heartbeat_back_addr': '172.21.15.60:6807/3490482162', 'heartbeat_front_addr': '172.21.15.60:6805/3490482162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '544f8a91-ce82-4855-85eb-ea814eb3d058', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6816', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6817', 'nonce': 1315554966}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6818', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6819', 'nonce': 1315554966}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6822', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6823', 'nonce': 1315554966}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6820', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6821', 'nonce': 1315554966}]}, 'public_addr': '172.21.15.60:6817/1315554966', 'cluster_addr': '172.21.15.60:6819/1315554966', 'heartbeat_back_addr': '172.21.15.60:6823/1315554966', 'heartbeat_front_addr': '172.21.15.60:6821/1315554966', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.514872+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.912129+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.745154+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:16.758871+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.287462+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:18.031636+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.055855+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.524609+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 2, 'snaps': [{'begin': 2, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T23:25:36.812 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:25:36.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:36.810+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:36.910 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:36.910+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:36.916 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:36.917 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:36.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:25:37.012 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:37.011+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:37.077 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:37.077 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:37.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:25:37.113 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:37.113+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:37.194 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:37.194 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:37.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:25:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-09-21T23:25:37.214 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:37.214+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:37.227 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:37.227 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:25:37.241 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-21T23:25:37.207034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":234692,"kb_used_data":1824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754740028,"statfs":{"total":773094113280,"available":772853788672,"internally_reserved":0,"allocated":1867776,"data_stored":1783722,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.431112"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-21T23:25:33.842287+0000","last_change":"2024-09-21T23:25:30.180125+0000","last_active":"2024-09-21T23:25:33.842287+0000","last_peered":"2024-09-21T23:25:33.842287+0000","last_clean":"2024-09-21T23:25:33.842287+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:25:33.842287+0000","last_undegraded":"2024-09-21T23:25:33.842287+0000","last_fullsized":"2024-09-21T23:25:33.842287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:17.515296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033759699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-21T23:25:33.841502+0000","last_change":"2024-09-21T23:25:30.189119+0000","last_active":"2024-09-21T23:25:33.841502+0000","last_peered":"2024-09-21T23:25:33.841502+0000","last_clean":"2024-09-21T23:25:33.841502+0000","last_became_active":"2024-09-21T23:25:28.165264+0000","last_became_peered":"2024-09-21T23:25:28.165264+0000","last_unstale":"2024-09-21T23:25:33.841502+0000","last_undegraded":"2024-09-21T23:25:33.841502+0000","last_fullsized":"2024-09-21T23:25:33.841502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:39:53.806934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053117699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-21T23:25:34.502341+0000","last_change":"2024-09-21T23:25:30.263565+0000","last_active":"2024-09-21T23:25:34.502341+0000","last_peered":"2024-09-21T23:25:34.502341+0000","last_clean":"2024-09-21T23:25:34.502341+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:25:34.502341+0000","last_undegraded":"2024-09-21T23:25:34.502341+0000","last_fullsized":"2024-09-21T23:25:34.502341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:46:37.545348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042085499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-21T23:25:33.841496+0000","last_change":"2024-09-21T23:25:30.189121+0000","last_active":"2024-09-21T23:25:33.841496+0000","last_peered":"2024-09-21T23:25:33.841496+0000","last_clean":"2024-09-21T23:25:33.841496+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:25:33.841496+0000","last_undegraded":"2024-09-21T23:25:33.841496+0000","last_fullsized":"2024-09-21T23:25:33.841496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:00:22.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054857199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-21T23:25:35.441220+0000","last_change":"2024-09-21T23:25:30.186984+0000","last_active":"2024-09-21T23:25:35.441220+0000","last_peered":"2024-09-21T23:25:35.441220+0000","last_clean":"2024-09-21T23:25:35.441220+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:25:35.441220+0000","last_undegraded":"2024-09-21T23:25:35.441220+0000","last_fullsized":"2024-09-21T23:25:35.441220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:01:38.740017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095847300000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-21T23:25:33.842161+0000","last_change":"2024-09-21T23:25:30.263343+0000","last_active":"2024-09-21T23:25:33.842161+0000","last_peered":"2024-09-21T23:25:33.842161+0000","last_clean":"2024-09-21T23:25:33.842161+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:25:33.842161+0000","last_undegraded":"2024-09-21T23:25:33.842161+0000","last_fullsized":"2024-09-21T23:25:33.842161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:50:16.699815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044148899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-21T23:25:34.502336+0000","last_change":"2024-09-21T23:25:30.263501+0000","last_active":"2024-09-21T23:25:34.502336+0000","last_peered":"2024-09-21T23:25:34.502336+0000","last_clean":"2024-09-21T23:25:34.502336+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:25:34.502336+0000","last_undegraded":"2024-09-21T23:25:34.502336+0000","last_fullsized":"2024-09-21T23:25:34.502336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.155893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036082099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-21T23:25:35.441216+0000","last_change":"2024-09-21T23:25:30.186477+0000","last_active":"2024-09-21T23:25:35.441216+0000","last_peered":"2024-09-21T23:25:35.441216+0000","last_clean":"2024-09-21T23:25:35.441216+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:25:35.441216+0000","last_undegraded":"2024-09-21T23:25:35.441216+0000","last_fullsized":"2024-09-21T23:25:35.441216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:32:47.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044681799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":172,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-21T23:25:34.502315+0000","last_change":"2024-09-21T23:25:21.135362+0000","last_active":"2024-09-21T23:25:34.502315+0000","last_peered":"2024-09-21T23:25:34.502315+0000","last_clean":"2024-09-21T23:25:34.502315+0000","last_became_active":"2024-09-21T23:25:21.135038+0000","last_became_peered":"2024-09-21T23:25:21.135038+0000","last_unstale":"2024-09-21T23:25:34.502315+0000","last_undegraded":"2024-09-21T23:25:34.502315+0000","last_fullsized":"2024-09-21T23:25:34.502315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:42:58.183351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":278528,"data_stored":667103,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":76754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31400,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":76735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":76754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":667103,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":76735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:25:37.242 INFO:tasks.thrashosds.thrasher:Setting 2.3 to [1, 3, 7, 5] 2024-09-21T23:25:37.242 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.3', '1', '3', '7', '5'] 2024-09-21T23:25:37.242 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.3 1 3 7 5 2024-09-21T23:25:37.315 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:37.315+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:37.321 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:37.321 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:37.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:25:37.416 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:37.415+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:37.479 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:37.480 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:37.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:25:37.516 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:37.516+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:37.617 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:37.617+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:37.636 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:37.636 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:37.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:25:37.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:37.730+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:37.793 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:37.793 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:37.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:25:37.819 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:37.819+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:37.920 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:37.919+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:37.948 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:37.949 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:37.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:25:38.020 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:38.020+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:38.105 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:38.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:25:38.157 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:38.157+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:set 2.3 pg_upmap_items mapping to [1->3,7->5] 2024-09-21T23:25:38.222 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:38.222+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:38.223 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:38.223 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:38.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:25:38.323 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:38.323+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:38.354 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:38.354 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:38.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:25:38.447 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:38.447+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:38.512 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:38.512 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:38.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:25:38.524 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:38.524+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:38.625 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:38.624+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:38.668 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:38.668 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:38.677 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:25:38.700 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:25:38.761 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:38.761+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:38.825 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:38.825 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:38.826 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:38.825+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:38.831 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:25:38.831 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:25:38.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:25:38.848 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:25:38.848 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:25:38.864 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:25:38.864 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:25:38.880 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:25:38.881 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:25:38.897 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:25:38.897 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:25:38.915 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:25:38.915 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:25:38.926 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:38.926+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:25:38.932 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:25:38.949 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:25:38.949 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:25:38.982 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:38.982 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:38.982 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:38.982 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:38.982 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:38.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:25:39.027 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:39.027+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:39.128 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:39.128+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:39.140 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:39.140 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:25:39.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:25:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:25:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:25:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:25:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 10.976525327999999, 2024-09-21T23:25:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0080986310000000002, 2024-09-21T23:25:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:25:39.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163709+0000", 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163727+0000", 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7552000000000001e-05 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163744+0000", 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7541999999999999e-05 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163757+0000", 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2855e-05 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164069+0000", 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031179899999999997 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164084+0000", 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5265e-05 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167127+0000", 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030427029999999999 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167317+0000", 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000190006 2024-09-21T23:25:39.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167465+0000", 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014824 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.168309+0000", 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084350899999999999 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171730+0000", 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034217620000000001 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171770+0000", 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0074999999999998e-05 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171802+0000", 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1563999999999998e-05 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.147 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:1 2.3 e19/18 ondisk, result = 0)", 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 10.968583081, 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016281800000000001, 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6899999999999996e-07 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171651+0000", 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0309999999999998e-06 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171665+0000", 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.309e-05 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171672+0000", 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6070000000000002e-06 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171716+0000", 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4344000000000001e-05 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171727+0000", 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0488e-05 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171808+0000", 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1489000000000005e-05 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.151 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 10.965807638999999, 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023803299999999999, 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:39.152 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 2 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174422+0000", 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4300000000000001e-06 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174429+0000", 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1659999999999999e-06 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174443+0000", 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4358e-05 2024-09-21T23:25:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174449+0000", 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2770000000000002e-06 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174540+0000", 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0209000000000003e-05 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174659+0000", 2024-09-21T23:25:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011959300000000001 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 10.965342638999999, 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016795100000000001, 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 3 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174887+0000", 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174892+0000", 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8180000000000003e-06 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174902+0000", 2024-09-21T23:25:39.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0095000000000001e-05 2024-09-21T23:25:39.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174911+0000", 2024-09-21T23:25:39.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0799999999999995e-06 2024-09-21T23:25:39.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174974+0000", 2024-09-21T23:25:39.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3027000000000005e-05 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.175054+0000", 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0011000000000004e-05 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 9.9718370079999996, 2024-09-21T23:25:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006147263, 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168394+0000", 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.956e-06 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168404+0000", 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0463e-05 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168423+0000", 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8715999999999998e-05 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168432+0000", 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.516e-06 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168529+0000", 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7073999999999999e-05 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168534+0000", 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.831e-06 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170224+0000", 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016893590000000001 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170332+0000", 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108346 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170445+0000", 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112642 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.171210+0000", 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076543800000000001 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174469+0000", 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003259086 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174506+0000", 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6465999999999999e-05 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174539+0000", 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3370000000000001e-05 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:5 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:25:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 9.9658949630000002, 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021298500000000001, 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174335+0000", 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.111e-06 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174341+0000", 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6239999999999996e-06 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174356+0000", 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4548e-05 2024-09-21T23:25:39.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174375+0000", 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8757e-05 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174457+0000", 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2702999999999997e-05 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174466+0000", 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.208e-06 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174547+0000", 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1033999999999997e-05 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 8.9561229240000007, 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040438610000000002, 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.170 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184109+0000", 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9069999999999999e-06 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184124+0000", 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5780000000000001e-05 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184159+0000", 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4106999999999997e-05 2024-09-21T23:25:39.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184178+0000", 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9908999999999999e-05 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184330+0000", 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015152300000000001 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184337+0000", 2024-09-21T23:25:39.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7920000000000004e-06 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186239+0000", 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019020129999999999 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186307+0000", 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8326999999999998e-05 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186459+0000", 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152099 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187261+0000", 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00080186199999999997 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188085+0000", 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00082417199999999997 2024-09-21T23:25:39.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188120+0000", 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5194999999999998e-05 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188150+0000", 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9175000000000001e-05 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.176 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:7 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 8.9522521879999992, 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000179505, 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.16e-07 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187985+0000", 2024-09-21T23:25:39.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3039999999999999e-06 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188000+0000", 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5097e-05 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188007+0000", 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1330000000000001e-06 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188074+0000", 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7514000000000004e-05 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188082+0000", 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3649999999999998e-06 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188156+0000", 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4176000000000003e-05 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.180 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.181 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.181 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.181 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:39.182 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:25:39.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T23:25:39.266 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:39.266+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:39.289 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:39.289 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:39.289 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:25:39.289 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:39.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:25:39.330 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:39.330+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:39.404 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:39.405 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:39.405 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:39.405 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:39.405 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:39.413 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:25:39.456 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:39.456+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:39.520 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:39.520 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:25:39.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:25:39.520 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-09-21T23:25:39.520 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:39.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:25:39.532 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:39.532+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:39.633 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:39.633+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:39.676 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:39.677 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:39.677 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:25:39.677 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:39.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:25:39.734 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:39.734+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:39.792 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:39.792 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:39.792 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:39.793 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:39.793 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:39.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:25:39.835 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:39.835+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:39.936 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:39.936+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:39.950 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:39.951 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:25:39.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:25:39.951 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:25:39.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.951 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:39.951 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:39.951 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.781893624999999, 2024-09-21T23:25:39.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090066050000000009, 2024-09-21T23:25:39.951 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.951 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.951 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 37 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7099999999999998e-07 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167009+0000", 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.92e-06 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167013+0000", 2024-09-21T23:25:39.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 2024-09-21T23:25:39.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167017+0000", 2024-09-21T23:25:39.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7280000000000001e-06 2024-09-21T23:25:39.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170778+0000", 2024-09-21T23:25:39.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037612399999999999 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170831+0000", 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2694000000000001e-05 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170970+0000", 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000138966 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.171555+0000", 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058538399999999999 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175872+0000", 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043172389999999996 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175969+0000", 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6844999999999999e-05 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.176012+0000", 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3356999999999998e-05 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.755788789, 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069023069999999999, 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.957 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:39.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:39.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0999999999999999e-07 2024-09-21T23:25:39.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193115+0000", 2024-09-21T23:25:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5700000000000001e-06 2024-09-21T23:25:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193129+0000", 2024-09-21T23:25:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4124000000000001e-05 2024-09-21T23:25:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193133+0000", 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0550000000000001e-06 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194144+0000", 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001010735 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194231+0000", 2024-09-21T23:25:39.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7665999999999997e-05 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194357+0000", 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125353 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194844+0000", 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000487305 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:39.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199964+0000", 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051204859999999996 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199992+0000", 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7902000000000002e-05 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.200013+0000", 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0601000000000001e-05 2024-09-21T23:25:39.963 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.734139715000001, 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0067494620000000003, 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:39.964 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:39.965 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 73 2024-09-21T23:25:39.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.965 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:39.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:39.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:39.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4900000000000001e-07 2024-09-21T23:25:39.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214818+0000", 2024-09-21T23:25:39.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7754000000000003e-05 2024-09-21T23:25:39.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214842+0000", 2024-09-21T23:25:39.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4552000000000001e-05 2024-09-21T23:25:39.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214846+0000", 2024-09-21T23:25:39.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9129999999999996e-06 2024-09-21T23:25:39.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215813+0000", 2024-09-21T23:25:39.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00096695899999999998 2024-09-21T23:25:39.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215904+0000", 2024-09-21T23:25:39.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0328000000000007e-05 2024-09-21T23:25:39.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:39.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216020+0000", 2024-09-21T23:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115838 2024-09-21T23:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217018+0000", 2024-09-21T23:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099836200000000003 2024-09-21T23:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221461+0000", 2024-09-21T23:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044435969999999997 2024-09-21T23:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221487+0000", 2024-09-21T23:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5806e-05 2024-09-21T23:25:39.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221509+0000", 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1903999999999999e-05 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:25:39.972 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.734045011999999, 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068139230000000004, 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 74 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:25:39.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214855+0000", 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-07 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214859+0000", 2024-09-21T23:25:39.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1389999999999997e-06 2024-09-21T23:25:39.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214871+0000", 2024-09-21T23:25:39.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1698999999999999e-05 2024-09-21T23:25:39.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214887+0000", 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6061000000000001e-05 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216359+0000", 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014720779999999999 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216439+0000", 2024-09-21T23:25:39.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0191999999999994e-05 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216579+0000", 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014029099999999999 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217129+0000", 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054932099999999997 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:39.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221559+0000", 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044299710000000004 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221627+0000", 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8631000000000001e-05 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221668+0000", 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0988000000000001e-05 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.979 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:39.979 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:39.979 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.717368826000001, 2024-09-21T23:25:39.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0077112509999999997, 2024-09-21T23:25:39.979 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.979 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.979 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:39.979 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.979 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 86 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:39.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5199999999999998e-07 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231533+0000", 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3879999999999998e-06 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231537+0000", 2024-09-21T23:25:39.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0010000000000003e-06 2024-09-21T23:25:39.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231540+0000", 2024-09-21T23:25:39.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0489999999999999e-06 2024-09-21T23:25:39.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233890+0000", 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023500249999999999 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233937+0000", 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6400999999999998e-05 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234015+0000", 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8343000000000005e-05 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:39.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235675+0000", 2024-09-21T23:25:39.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001660124 2024-09-21T23:25:39.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:39.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239138+0000", 2024-09-21T23:25:39.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034624109999999999 2024-09-21T23:25:39.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239200+0000", 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2698000000000002e-05 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239242+0000", 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1458999999999997e-05 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.717352557000002, 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010200097999999999, 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87 2024-09-21T23:25:39.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.987 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:39.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:39.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:39.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:39.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:39.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2700000000000001e-07 2024-09-21T23:25:39.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:39.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231588+0000", 2024-09-21T23:25:39.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0985000000000001e-05 2024-09-21T23:25:39.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231593+0000", 2024-09-21T23:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1059999999999999e-06 2024-09-21T23:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231606+0000", 2024-09-21T23:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2343e-05 2024-09-21T23:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234406+0000", 2024-09-21T23:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027999850000000001 2024-09-21T23:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234489+0000", 2024-09-21T23:25:39.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3051000000000002e-05 2024-09-21T23:25:39.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:39.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234631+0000", 2024-09-21T23:25:39.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285799999999999 2024-09-21T23:25:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235679+0000", 2024-09-21T23:25:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001047517 2024-09-21T23:25:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241644+0000", 2024-09-21T23:25:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059652129999999996 2024-09-21T23:25:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241704+0000", 2024-09-21T23:25:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9732999999999999e-05 2024-09-21T23:25:39.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:39.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241747+0000", 2024-09-21T23:25:39.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-05 2024-09-21T23:25:39.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.997 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:39.997 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:39.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.997 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:39.997 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:39.997 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.717286718, 2024-09-21T23:25:39.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010246534, 2024-09-21T23:25:39.998 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:39.998 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:39.998 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:39.998 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:39.998 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:39.998 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:39.998 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 88 2024-09-21T23:25:39.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:39.999 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:39.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:39.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:39.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:39.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:39.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:40.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:40.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0199999999999998e-07 2024-09-21T23:25:40.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231615+0000", 2024-09-21T23:25:40.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3439999999999999e-06 2024-09-21T23:25:40.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231619+0000", 2024-09-21T23:25:40.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0219999999999998e-06 2024-09-21T23:25:40.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231622+0000", 2024-09-21T23:25:40.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6989999999999999e-06 2024-09-21T23:25:40.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235337+0000", 2024-09-21T23:25:40.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037150030000000001 2024-09-21T23:25:40.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235385+0000", 2024-09-21T23:25:40.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7861999999999997e-05 2024-09-21T23:25:40.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235458+0000", 2024-09-21T23:25:40.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2596999999999996e-05 2024-09-21T23:25:40.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235978+0000", 2024-09-21T23:25:40.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052013600000000004 2024-09-21T23:25:40.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241799+0000", 2024-09-21T23:25:40.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058215289999999998 2024-09-21T23:25:40.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241836+0000", 2024-09-21T23:25:40.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6282999999999998e-05 2024-09-21T23:25:40.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241859+0000", 2024-09-21T23:25:40.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3757e-05 2024-09-21T23:25:40.007 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.008 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.008 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.008 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:40.008 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:40.008 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.559895534999999, 2024-09-21T23:25:40.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072206969999999999, 2024-09-21T23:25:40.009 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.009 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.009 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.009 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.009 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.009 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.009 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 112 2024-09-21T23:25:40.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.010 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:40.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:40.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:40.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.72e-07 2024-09-21T23:25:40.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389069+0000", 2024-09-21T23:25:40.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4875000000000005e-05 2024-09-21T23:25:40.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389079+0000", 2024-09-21T23:25:40.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0135e-05 2024-09-21T23:25:40.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389085+0000", 2024-09-21T23:25:40.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-09-21T23:25:40.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394006+0000", 2024-09-21T23:25:40.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049207920000000002 2024-09-21T23:25:40.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394052+0000", 2024-09-21T23:25:40.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5810999999999999e-05 2024-09-21T23:25:40.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394139+0000", 2024-09-21T23:25:40.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7130000000000001e-05 2024-09-21T23:25:40.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395407+0000", 2024-09-21T23:25:40.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012681260000000001 2024-09-21T23:25:40.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396139+0000", 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000731703 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396194+0000", 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5309999999999997e-05 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396225+0000", 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0589000000000001e-05 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.018 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.559788577999999, 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086923490000000003, 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.019 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 113 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389116+0000", 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.245e-06 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389126+0000", 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0295000000000001e-05 2024-09-21T23:25:40.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389132+0000", 2024-09-21T23:25:40.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0889999999999996e-06 2024-09-21T23:25:40.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394548+0000", 2024-09-21T23:25:40.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005415876 2024-09-21T23:25:40.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394642+0000", 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4531999999999995e-05 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394714+0000", 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1753000000000002e-05 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396233+0000", 2024-09-21T23:25:40.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015185299999999999 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397749+0000", 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015157770000000001 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397779+0000", 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0561000000000001e-05 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397803+0000", 2024-09-21T23:25:40.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4116000000000001e-05 2024-09-21T23:25:40.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.025 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.025 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:40.025 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:40.025 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.559756509, 2024-09-21T23:25:40.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.01139824, 2024-09-21T23:25:40.025 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.025 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.026 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.026 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 114 2024-09-21T23:25:40.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.026 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:40.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:40.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:40.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.01e-07 2024-09-21T23:25:40.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389146+0000", 2024-09-21T23:25:40.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5239999999999999e-06 2024-09-21T23:25:40.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389150+0000", 2024-09-21T23:25:40.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3989999999999997e-06 2024-09-21T23:25:40.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389153+0000", 2024-09-21T23:25:40.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.038e-06 2024-09-21T23:25:40.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395026+0000", 2024-09-21T23:25:40.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058731720000000003 2024-09-21T23:25:40.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395071+0000", 2024-09-21T23:25:40.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4388999999999998e-05 2024-09-21T23:25:40.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395141+0000", 2024-09-21T23:25:40.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0053000000000002e-05 2024-09-21T23:25:40.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397196+0000", 2024-09-21T23:25:40.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020555899999999999 2024-09-21T23:25:40.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400494+0000", 2024-09-21T23:25:40.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032972679999999999 2024-09-21T23:25:40.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400520+0000", 2024-09-21T23:25:40.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6423000000000001e-05 2024-09-21T23:25:40.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400541+0000", 2024-09-21T23:25:40.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0982999999999999e-05 2024-09-21T23:25:40.034 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.034 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.034 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.035 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:40.035 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:40.035 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.55973783, 2024-09-21T23:25:40.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011537824, 2024-09-21T23:25:40.035 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.035 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.035 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.035 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.035 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.036 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.036 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 115 2024-09-21T23:25:40.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.036 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:40.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:40.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:40.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4900000000000001e-07 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389197+0000", 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5204999999999999e-05 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389203+0000", 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0979999999999999e-06 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389207+0000", 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6600000000000001e-06 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395443+0000", 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062358099999999996 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395525+0000", 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1855999999999997e-05 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395669+0000", 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014472399999999999 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397204+0000", 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001534817 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400599+0000", 2024-09-21T23:25:40.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033944499999999998 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400655+0000", 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6647000000000002e-05 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400699+0000", 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4208000000000001e-05 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.559686377999999, 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011571580999999999, 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 116 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.042 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3099999999999999e-07 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389216+0000", 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4260000000000002e-06 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389223+0000", 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389225+0000", 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7760000000000002e-06 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396256+0000", 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070309020000000003 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396348+0000", 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1780000000000006e-05 2024-09-21T23:25:40.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396481+0000", 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000133101 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397208+0000", 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072716199999999995 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400737+0000", 2024-09-21T23:25:40.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035290220000000001 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400763+0000", 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5219999999999999e-05 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400785+0000", 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2045e-05 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.048 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:40.048 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:40.048 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.559669695, 2024-09-21T23:25:40.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011687858000000001, 2024-09-21T23:25:40.048 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.048 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.048 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.048 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.048 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.048 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.048 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 117 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3699999999999999e-07 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389232+0000", 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1789999999999998e-06 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389238+0000", 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209999999999999e-06 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389241+0000", 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.557e-06 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396792+0000", 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075506899999999997 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396838+0000", 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5639000000000001e-05 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396958+0000", 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119892 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397458+0000", 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049987600000000001 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400824+0000", 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033662560000000002 2024-09-21T23:25:40.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400880+0000", 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5779999999999998e-05 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400918+0000", 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8031000000000002e-05 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.053 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.547651220999999, 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006886924, 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 120 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401249+0000", 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.13e-07 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401286+0000", 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6773999999999997e-05 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401295+0000", 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9039999999999998e-06 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401299+0000", 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3250000000000001e-06 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402183+0000", 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000883758 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402264+0000", 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1080000000000003e-05 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402401+0000", 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013719700000000001 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403430+0000", 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010290119999999999 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408045+0000", 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046146659999999999 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408095+0000", 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0898e-05 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408135+0000", 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9697000000000003e-05 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.547592746999999, 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069210920000000002, 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.059 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 121 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3200000000000001e-07 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401341+0000", 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4236999999999998e-05 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401346+0000", 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.04e-06 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401350+0000", 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6399999999999999e-06 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402733+0000", 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013825790000000001 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402816+0000", 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3461000000000006e-05 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402941+0000", 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012518300000000001 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403533+0000", 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000592071 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408174+0000", 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046404209999999996 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408204+0000", 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0147000000000001e-05 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408228+0000", 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3981e-05 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.064 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.11144985, 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064924400000000004, 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.065 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 139 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0800000000000001e-07 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837452+0000", 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7209999999999999e-06 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837455+0000", 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2279999999999998e-06 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837458+0000", 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8200000000000001e-06 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842187+0000", 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047298460000000002 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842233+0000", 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6056000000000001e-05 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842304+0000", 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0862e-05 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843291+0000", 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00098632100000000007 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843894+0000", 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060369199999999997 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843922+0000", 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7429000000000001e-05 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843942+0000", 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0256999999999999e-05 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:40.070 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.111437906999999, 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066155550000000004, 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 140 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:40.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1e-07 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837517+0000", 2024-09-21T23:25:40.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5034000000000002e-05 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837522+0000", 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8520000000000003e-06 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837525+0000", 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4139999999999998e-06 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842562+0000", 2024-09-21T23:25:40.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050366279999999996 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842615+0000", 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3675999999999999e-05 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842715+0000", 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9990999999999999e-05 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843873+0000", 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011573 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844026+0000", 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015368900000000001 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844058+0000", 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1449999999999999e-05 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844077+0000", 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9310999999999999e-05 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.111365714000002, 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068324850000000001, 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 141 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.22e-07 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837539+0000", 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.023e-06 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837543+0000", 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6490000000000001e-06 2024-09-21T23:25:40.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837546+0000", 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.382e-06 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843479+0000", 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059328189999999998 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843519+0000", 2024-09-21T23:25:40.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0071000000000003e-05 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843600+0000", 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1376000000000002e-05 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844137+0000", 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053640299999999997 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844319+0000", 2024-09-21T23:25:40.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000181893 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844346+0000", 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.781e-05 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844366+0000", 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9737000000000001e-05 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 17.496032019000001, 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006594145, 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 158 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452868+0000", 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9500000000000003e-07 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452938+0000", 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9598999999999996e-05 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452950+0000", 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2971e-05 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452960+0000", 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.448e-06 2024-09-21T23:25:40.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456217+0000", 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003257378 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456557+0000", 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033989399999999998 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456703+0000", 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000145599 2024-09-21T23:25:40.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459339+0000", 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026365529999999998 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459386+0000", 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6640000000000001e-05 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459428+0000", 2024-09-21T23:25:40.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2148999999999999e-05 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459462+0000", 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3419e-05 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 17.495927644999998, 2024-09-21T23:25:40.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0065672760000000004, 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 159 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5400000000000001e-07 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452977+0000", 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.301e-06 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452984+0000", 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1840000000000002e-06 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452989+0000", 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5219999999999998e-06 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457324+0000", 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043348780000000003 2024-09-21T23:25:40.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457416+0000", 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1824999999999996e-05 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457574+0000", 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015810999999999999 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459376+0000", 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001801649 2024-09-21T23:25:40.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459467+0000", 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1367999999999997e-05 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459507+0000", 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9576000000000002e-05 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459539+0000", 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2308999999999998e-05 2024-09-21T23:25:40.092 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.093 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:40.093 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:40.093 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:40.094 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:40.094+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:40.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:25:40.194 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:40.194+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:40.214 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:40.214 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:40.214 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:40.215 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:40.223 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-09-21T23:25:40.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:25:40.295 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:40.295+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:40.332 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:40.333 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:40.333 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:40.333 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:40.333 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:40.341 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:25:40.395 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:40.395+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:40.450 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:40.450 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:40.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:40.450 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:40.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.450 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-09-21T23:25:40.450 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:40.450 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.312033066000001, 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036127939999999999, 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:40.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3500000000000002e-07 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136608+0000", 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.182e-06 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136618+0000", 2024-09-21T23:25:40.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9199999999999999e-06 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136628+0000", 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4390000000000006e-06 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136692+0000", 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3979000000000005e-05 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136716+0000", 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4391000000000001e-05 2024-09-21T23:25:40.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140171+0000", 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034554910000000002 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140214+0000", 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2057000000000001e-05 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.454 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'18)", 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.277514903, 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043957730000000004, 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171120+0000", 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6700000000000003e-07 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171125+0000", 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.451e-06 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171138+0000", 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2541000000000001e-05 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171143+0000", 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4659999999999998e-06 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171187+0000", 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4014000000000002e-05 2024-09-21T23:25:40.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171200+0000", 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3108999999999999e-05 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175479+0000", 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042784570000000003 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175515+0000", 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5967999999999999e-05 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.458 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:58 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.254762607, 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003032851, 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193872+0000", 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9400000000000004e-07 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193952+0000", 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9791999999999998e-05 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193962+0000", 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0034000000000001e-05 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193973+0000", 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0733e-05 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194015+0000", 2024-09-21T23:25:40.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2911999999999998e-05 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194031+0000", 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5082999999999999e-05 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196869+0000", 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002838808 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196904+0000", 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4694999999999999e-05 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:59 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.254176941000001, 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053675709999999998, 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7100000000000002e-07 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194462+0000", 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.668e-06 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194469+0000", 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1670000000000002e-06 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194475+0000", 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-06 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194513+0000", 2024-09-21T23:25:40.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7391999999999997e-05 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194531+0000", 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7870999999999999e-05 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199791+0000", 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052604119999999999 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199824+0000", 2024-09-21T23:25:40.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3340000000000003e-05 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:60 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'38, mlcod=16'36)", 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.248096245999999, 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003614562, 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.468 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:40.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:40.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200539+0000", 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5799999999999999e-06 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200551+0000", 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1816e-05 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200578+0000", 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6772999999999999e-05 2024-09-21T23:25:40.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200590+0000", 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2303000000000001e-05 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200715+0000", 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012523499999999999 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200731+0000", 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5690000000000001e-05 2024-09-21T23:25:40.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204121+0000", 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003390363 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204152+0000", 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0802000000000001e-05 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:70 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'39)", 2024-09-21T23:25:40.472 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.238040591000001, 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033896579999999998, 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:40.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210594+0000", 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1099999999999995e-07 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210600+0000", 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4389999999999997e-06 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210613+0000", 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.243e-05 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210618+0000", 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0939999999999998e-06 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210843+0000", 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022493200000000001 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210858+0000", 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5109000000000001e-05 2024-09-21T23:25:40.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213938+0000", 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030796500000000002 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213983+0000", 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5293000000000002e-05 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:72 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'46, mlcod=16'43)", 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.233095814999999, 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029102279999999999, 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215539+0000", 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4199999999999995e-07 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215589+0000", 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0488999999999997e-05 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215598+0000", 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2830000000000007e-06 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215610+0000", 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1134999999999999e-05 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215657+0000", 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7658999999999999e-05 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215679+0000", 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1526999999999999e-05 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218410+0000", 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027316939999999998 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218448+0000", 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7799000000000002e-05 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:73 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'47, mlcod=16'43)", 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.232486468000001, 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051882020000000003, 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216148+0000", 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7100000000000001e-07 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216218+0000", 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0313000000000003e-05 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216227+0000", 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9959999999999999e-06 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216232+0000", 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5340000000000001e-06 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216267+0000", 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4739000000000001e-05 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216281+0000", 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4345000000000001e-05 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221290+0000", 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050088989999999998 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221336+0000", 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5704999999999997e-05 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:74 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'48, mlcod=16'43)", 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.231960525000002, 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046922939999999996, 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216674+0000", 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9900000000000001e-07 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216679+0000", 2024-09-21T23:25:40.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.502e-06 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216687+0000", 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1790000000000008e-06 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216691+0000", 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7230000000000002e-06 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216737+0000", 2024-09-21T23:25:40.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6087000000000001e-05 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216758+0000", 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0434000000000001e-05 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221338+0000", 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045804569999999996 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221366+0000", 2024-09-21T23:25:40.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7413e-05 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:86 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'56, mlcod=16'50)", 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.214538591, 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049055269999999998, 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234096+0000", 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234099+0000", 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3890000000000001e-06 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234107+0000", 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2349999999999999e-06 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234112+0000", 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0629999999999999e-06 2024-09-21T23:25:40.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234143+0000", 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1131999999999999e-05 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234155+0000", 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2140000000000001e-05 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.238972+0000", 2024-09-21T23:25:40.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048167770000000004 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.239001+0000", 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9164000000000001e-05 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:87 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'57, mlcod=16'52)", 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.213859215999999, 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067179910000000004, 2024-09-21T23:25:40.493 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0299999999999998e-07 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234813+0000", 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7531999999999999e-05 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234826+0000", 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2617e-05 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234831+0000", 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1229999999999999e-06 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234869+0000", 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8513999999999998e-05 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234881+0000", 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2148e-05 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241465+0000", 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065837329999999996 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241493+0000", 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7521e-05 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.497 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:88 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.213061368000002, 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059344530000000001, 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235573+0000", 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0100000000000004e-07 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235578+0000", 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1590000000000001e-06 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235590+0000", 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1229e-05 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235595+0000", 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0889999999999999e-06 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235943+0000", 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000348227 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235957+0000", 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4419999999999999e-05 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241494+0000", 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055371070000000003 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241507+0000", 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2520999999999999e-05 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:113 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'75, mlcod=16'71)", 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.053830090000002, 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028543489999999999, 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8199999999999999e-07 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394809+0000", 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394821+0000", 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2024999999999999e-05 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394827+0000", 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0020000000000003e-06 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394862+0000", 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4811000000000003e-05 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.504 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394873+0000", 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.058e-05 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397631+0000", 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027581060000000002 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397658+0000", 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7073999999999999e-05 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.505 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'71)", 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.053418677, 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005160085, 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.506 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395216+0000", 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4999999999999998e-07 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395221+0000", 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2290000000000002e-06 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395229+0000", 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2369999999999992e-06 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395233+0000", 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1840000000000001e-06 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395260+0000", 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.728e-05 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395272+0000", 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1270000000000001e-05 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400341+0000", 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050691809999999999 2024-09-21T23:25:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400375+0000", 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4254000000000001e-05 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'72)", 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.052870156000001, 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046274580000000001, 2024-09-21T23:25:40.510 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395829+0000", 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4618999999999999e-05 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395836+0000", 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9090000000000003e-06 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395839+0000", 2024-09-21T23:25:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.371e-06 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395868+0000", 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9328000000000001e-05 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395880+0000", 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1501000000000001e-05 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400376+0000", 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044966340000000002 2024-09-21T23:25:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400391+0000", 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4676e-05 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:116 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'78, mlcod=16'73)", 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.052065609, 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038400970000000002, 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396569+0000", 2024-09-21T23:25:40.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1100000000000001e-07 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396573+0000", 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5569999999999997e-06 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396579+0000", 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1120000000000001e-06 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396587+0000", 2024-09-21T23:25:40.516 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396630+0000", 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2991000000000002e-05 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396642+0000", 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1481000000000001e-05 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400392+0000", 2024-09-21T23:25:40.517 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037507830000000002 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400408+0000", 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5863e-05 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:117 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'79, mlcod=16'73)", 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.051563266999999, 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033512120000000001, 2024-09-21T23:25:40.518 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.519 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7e-07 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397075+0000", 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7809999999999999e-06 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397082+0000", 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397085+0000", 2024-09-21T23:25:40.520 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7139999999999999e-06 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397130+0000", 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.498e-05 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397142+0000", 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1838000000000001e-05 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400409+0000", 2024-09-21T23:25:40.521 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032666349999999999 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400422+0000", 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2866e-05 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.046739594000002, 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034761240000000001, 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.522 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.523 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401895+0000", 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7400000000000003e-07 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401963+0000", 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8701000000000005e-05 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401972+0000", 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9530000000000005e-06 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.524 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401977+0000", 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1579999999999998e-06 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402009+0000", 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2512000000000003e-05 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402022+0000", 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3220999999999999e-05 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.525 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405337+0000", 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033150300000000001 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405370+0000", 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2975000000000002e-05 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:40.526 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.046133523999998, 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054168369999999999, 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402501+0000", 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2699999999999999e-07 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402589+0000", 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899000000000007e-05 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402599+0000", 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0312000000000001e-05 2024-09-21T23:25:40.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402605+0000", 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9689999999999999e-06 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402648+0000", 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3210000000000001e-05 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402664+0000", 2024-09-21T23:25:40.529 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5347999999999999e-05 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407887+0000", 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052237919999999997 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407917+0000", 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9779999999999999e-05 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 19.045604142999998, 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049049180000000003, 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.531 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4099999999999999e-07 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403098+0000", 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7464000000000003e-05 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.532 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403106+0000", 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6629999999999999e-06 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403111+0000", 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403137+0000", 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6349000000000001e-05 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403148+0000", 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.077e-05 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407920+0000", 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047716590000000001 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407935+0000", 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4783999999999999e-05 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.534 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.535 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.535 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:40.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:25:40.536 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:40.536+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:40.636 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:40.636+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:40.644 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:40.644 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:40.644 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:40.644 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:40.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:25:40.738 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:40.738+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:40.800 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:40.801 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:40.801 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:40.801 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:40.801 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:40.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:25:40.837 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:40.837+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:40.938 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:40.938+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:40.956 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 11.78630396, 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003527526, 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170743+0000", 2024-09-21T23:25:40.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.08e-07 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170750+0000", 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800000000000001e-06 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170767+0000", 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6923999999999999e-05 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.959 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170788+0000", 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0446e-05 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170849+0000", 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1409999999999996e-05 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170881+0000", 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2075e-05 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.960 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174218+0000", 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033371070000000002 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174270+0000", 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1375999999999997e-05 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 10.770314466, 2024-09-21T23:25:40.961 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011783570000000001, 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:40.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7999999999999996e-07 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186750+0000", 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7444000000000001e-05 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186834+0000", 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3974999999999995e-05 2024-09-21T23:25:40.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186876+0000", 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2796000000000002e-05 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186999+0000", 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012303800000000001 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187036+0000", 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6897000000000003e-05 2024-09-21T23:25:40.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187869+0000", 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00083215199999999996 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187910+0000", 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1575000000000001e-05 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:40.965 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:40.966 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:40.966 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:25:40.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:25:41.051 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:41.051+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:41.118 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:41.118 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:41.118 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:41.118 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:41.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:25:41.134 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:25:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:25:41.151 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:25:41.152 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:25:41.152 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:41.151+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:41.168 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:25:41.169 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:25:41.185 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:25:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:25:41.203 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:25:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:25:41.220 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:25:41.220 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:25:41.240 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:25:41.240 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:25:41.240 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:41.240+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:41.242 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:41.242 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:41.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:25:41.257 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:25:41.258 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:25:41.342 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:41.341+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:41.364 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:41.365 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:41.373 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:25:41.443 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:41.442+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:41.480 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:41.480 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:41.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:25:41.543 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:41.543+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:41.598 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:41.598 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:41.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:25:41.650 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:41.650+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:41.714 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:41.714 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:41.722 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:25:41.763 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:41.763+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:41.843 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:41.843 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:41.845 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:41.845+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:41.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:25:41.946 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:41.946+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:42.001 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:42.001 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:42.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:25:42.047 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:42.047+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:42.148 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:42.148+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:42.158 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:42.158 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:42.167 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:25:42.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T23:25:42.282 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:42.282+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:42.316 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:42.316 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:42.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:25:42.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:42.351+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:42.433 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:42.433 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:42.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:25:42.485 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:42.485+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:42.548 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:42.548 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:42.553 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:42.553+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:42.557 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:25:42.655 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:42.655+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:42.705 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:42.705 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:42.714 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:25:42.756 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:42.756+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:42.856 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:42.856+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:42.861 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:42.861 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:42.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:25:42.958 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:42.958+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:42.978 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:42.978 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:42.987 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:25:43.059 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:43.059+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:43.136 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:43.136 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:43.144 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:25:43.195 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:43.195+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:43.747 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:25:43.747 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:25:43.748 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T23:25:43.748 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 1.000000 2024-09-21T23:25:43.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-21T23:25:43.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:25:43.751 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-09-21T23:25:43.751 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:43.751 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:43.752 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:25:43.752 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:43.750+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:43.850 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:43.850+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:43.911 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:43.911 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:43.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:25:43.942 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:25:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:25:43.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:43.951+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:43.973 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:25:43.974 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:25:43.990 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:25:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:25:44.007 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:25:44.007 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:25:44.024 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:25:44.024 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:25:44.031 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:44.032 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:44.032 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:44.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:25:44.041 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:25:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:25:44.058 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:25:44.059 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:25:44.059 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:44.058+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:44.076 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:25:44.076 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:25:44.150 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:44.150 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:44.150 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:44.152 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:44.152+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:44.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:25:44.241 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T23:25:44.253 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:44.253+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:44.312 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-09-21T23:25:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.173196147999999, 2024-09-21T23:25:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036127939999999999, 2024-09-21T23:25:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:44.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3500000000000002e-07 2024-09-21T23:25:44.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136608+0000", 2024-09-21T23:25:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.182e-06 2024-09-21T23:25:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136618+0000", 2024-09-21T23:25:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9199999999999999e-06 2024-09-21T23:25:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136628+0000", 2024-09-21T23:25:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4390000000000006e-06 2024-09-21T23:25:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.319 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136692+0000", 2024-09-21T23:25:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3979000000000005e-05 2024-09-21T23:25:44.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136716+0000", 2024-09-21T23:25:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4391000000000001e-05 2024-09-21T23:25:44.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140171+0000", 2024-09-21T23:25:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034554910000000002 2024-09-21T23:25:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140214+0000", 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2057000000000001e-05 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'18)", 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.138677985000001, 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043957730000000004, 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.323 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171120+0000", 2024-09-21T23:25:44.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6700000000000003e-07 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171125+0000", 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.451e-06 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171138+0000", 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2541000000000001e-05 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171143+0000", 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4659999999999998e-06 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171187+0000", 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4014000000000002e-05 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171200+0000", 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3108999999999999e-05 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175479+0000", 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042784570000000003 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175515+0000", 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5967999999999999e-05 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:58 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:44.327 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.115925689000001, 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003032851, 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193872+0000", 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9400000000000004e-07 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193952+0000", 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9791999999999998e-05 2024-09-21T23:25:44.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193962+0000", 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0034000000000001e-05 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193973+0000", 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0733e-05 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194015+0000", 2024-09-21T23:25:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2911999999999998e-05 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194031+0000", 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5082999999999999e-05 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196869+0000", 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002838808 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196904+0000", 2024-09-21T23:25:44.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4694999999999999e-05 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:59 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.115340023000002, 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053675709999999998, 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7100000000000002e-07 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194462+0000", 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.668e-06 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194469+0000", 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1670000000000002e-06 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194475+0000", 2024-09-21T23:25:44.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-06 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194513+0000", 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7391999999999997e-05 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194531+0000", 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7870999999999999e-05 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199791+0000", 2024-09-21T23:25:44.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052604119999999999 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199824+0000", 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3340000000000003e-05 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:60 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'38, mlcod=16'36)", 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.109259328, 2024-09-21T23:25:44.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003614562, 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200539+0000", 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5799999999999999e-06 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200551+0000", 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1816e-05 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200578+0000", 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6772999999999999e-05 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200590+0000", 2024-09-21T23:25:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2303000000000001e-05 2024-09-21T23:25:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200715+0000", 2024-09-21T23:25:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012523499999999999 2024-09-21T23:25:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200731+0000", 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5690000000000001e-05 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204121+0000", 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003390363 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204152+0000", 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0802000000000001e-05 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.340 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:70 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'39)", 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.099203673000002, 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033896579999999998, 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210594+0000", 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1099999999999995e-07 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210600+0000", 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4389999999999997e-06 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210613+0000", 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.243e-05 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210618+0000", 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0939999999999998e-06 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210843+0000", 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022493200000000001 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210858+0000", 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5109000000000001e-05 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213938+0000", 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030796500000000002 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.344 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213983+0000", 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5293000000000002e-05 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:72 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'46, mlcod=16'43)", 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.094258897, 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029102279999999999, 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.345 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215539+0000", 2024-09-21T23:25:44.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4199999999999995e-07 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215589+0000", 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0488999999999997e-05 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215598+0000", 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2830000000000007e-06 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215610+0000", 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1134999999999999e-05 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215657+0000", 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7658999999999999e-05 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215679+0000", 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1526999999999999e-05 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.348 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218410+0000", 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027316939999999998 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218448+0000", 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7799000000000002e-05 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:73 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'47, mlcod=16'43)", 2024-09-21T23:25:44.349 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.093649549999999, 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051882020000000003, 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216148+0000", 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7100000000000001e-07 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216218+0000", 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0313000000000003e-05 2024-09-21T23:25:44.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216227+0000", 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9959999999999999e-06 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216232+0000", 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5340000000000001e-06 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216267+0000", 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4739000000000001e-05 2024-09-21T23:25:44.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216281+0000", 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4345000000000001e-05 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221290+0000", 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050088989999999998 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221336+0000", 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5704999999999997e-05 2024-09-21T23:25:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:74 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'48, mlcod=16'43)", 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.093123606999999, 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046922939999999996, 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.354 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216674+0000", 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9900000000000001e-07 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.355 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216679+0000", 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.502e-06 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216687+0000", 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1790000000000008e-06 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216691+0000", 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7230000000000002e-06 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216737+0000", 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6087000000000001e-05 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216758+0000", 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0434000000000001e-05 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221338+0000", 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045804569999999996 2024-09-21T23:25:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221366+0000", 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7413e-05 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:86 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'56, mlcod=16'50)", 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.075701673000001, 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049055269999999998, 2024-09-21T23:25:44.358 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.359 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234096+0000", 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234099+0000", 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3890000000000001e-06 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234107+0000", 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2349999999999999e-06 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234112+0000", 2024-09-21T23:25:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0629999999999999e-06 2024-09-21T23:25:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234143+0000", 2024-09-21T23:25:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1131999999999999e-05 2024-09-21T23:25:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.361 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234155+0000", 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2140000000000001e-05 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.238972+0000", 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048167770000000004 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.239001+0000", 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9164000000000001e-05 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:87 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'57, mlcod=16'52)", 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.075022298, 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067179910000000004, 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.363 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0299999999999998e-07 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.364 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234813+0000", 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7531999999999999e-05 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234826+0000", 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2617e-05 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234831+0000", 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1229999999999999e-06 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234869+0000", 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8513999999999998e-05 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234881+0000", 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2148e-05 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241465+0000", 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065837329999999996 2024-09-21T23:25:44.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241493+0000", 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7521e-05 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:88 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 23.074224449999999, 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059344530000000001, 2024-09-21T23:25:44.367 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.368 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235573+0000", 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0100000000000004e-07 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235578+0000", 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1590000000000001e-06 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235590+0000", 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1229e-05 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.369 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235595+0000", 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0889999999999999e-06 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235943+0000", 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000348227 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235957+0000", 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4419999999999999e-05 2024-09-21T23:25:44.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241494+0000", 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055371070000000003 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241507+0000", 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2520999999999999e-05 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:113 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'75, mlcod=16'71)", 2024-09-21T23:25:44.371 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 22.914993171999999, 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028543489999999999, 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:44.372 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8199999999999999e-07 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394809+0000", 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.373 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394821+0000", 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2024999999999999e-05 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394827+0000", 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0020000000000003e-06 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394862+0000", 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4811000000000003e-05 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394873+0000", 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.058e-05 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397631+0000", 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027581060000000002 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397658+0000", 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7073999999999999e-05 2024-09-21T23:25:44.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'71)", 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 22.914581759000001, 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005160085, 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.376 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395216+0000", 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4999999999999998e-07 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395221+0000", 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2290000000000002e-06 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395229+0000", 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2369999999999992e-06 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395233+0000", 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1840000000000001e-06 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.378 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395260+0000", 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.728e-05 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395272+0000", 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1270000000000001e-05 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400341+0000", 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050691809999999999 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.379 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400375+0000", 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4254000000000001e-05 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'72)", 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 22.914033238000002, 2024-09-21T23:25:44.380 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046274580000000001, 2024-09-21T23:25:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-09-21T23:25:44.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395829+0000", 2024-09-21T23:25:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4618999999999999e-05 2024-09-21T23:25:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.384 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395836+0000", 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9090000000000003e-06 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395839+0000", 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.371e-06 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395868+0000", 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9328000000000001e-05 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395880+0000", 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1501000000000001e-05 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400376+0000", 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044966340000000002 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400391+0000", 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4676e-05 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.386 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:116 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'78, mlcod=16'73)", 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 22.913228691, 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038400970000000002, 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.387 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396569+0000", 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1100000000000001e-07 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.388 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396573+0000", 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5569999999999997e-06 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396579+0000", 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1120000000000001e-06 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396587+0000", 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.389 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396630+0000", 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2991000000000002e-05 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396642+0000", 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1481000000000001e-05 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400392+0000", 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037507830000000002 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400408+0000", 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5863e-05 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:117 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'79, mlcod=16'73)", 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 22.912726349, 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033512120000000001, 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.391 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:44.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7e-07 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397075+0000", 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7809999999999999e-06 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397082+0000", 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397085+0000", 2024-09-21T23:25:44.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7139999999999999e-06 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397130+0000", 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.498e-05 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397142+0000", 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1838000000000001e-05 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400409+0000", 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032666349999999999 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400422+0000", 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2866e-05 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 22.907902675999999, 2024-09-21T23:25:44.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034761240000000001, 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401895+0000", 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7400000000000003e-07 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401963+0000", 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8701000000000005e-05 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401972+0000", 2024-09-21T23:25:44.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9530000000000005e-06 2024-09-21T23:25:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401977+0000", 2024-09-21T23:25:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1579999999999998e-06 2024-09-21T23:25:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402009+0000", 2024-09-21T23:25:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2512000000000003e-05 2024-09-21T23:25:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.399 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402022+0000", 2024-09-21T23:25:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3220999999999999e-05 2024-09-21T23:25:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.400 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405337+0000", 2024-09-21T23:25:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033150300000000001 2024-09-21T23:25:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405370+0000", 2024-09-21T23:25:44.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2975000000000002e-05 2024-09-21T23:25:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-09-21T23:25:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 22.907296605999999, 2024-09-21T23:25:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054168369999999999, 2024-09-21T23:25:44.402 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402501+0000", 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2699999999999999e-07 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402589+0000", 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899000000000007e-05 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402599+0000", 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0312000000000001e-05 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402605+0000", 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9689999999999999e-06 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402648+0000", 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3210000000000001e-05 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402664+0000", 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5347999999999999e-05 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407887+0000", 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052237919999999997 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407917+0000", 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9779999999999999e-05 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-09-21T23:25:44.406 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 22.906767224999999, 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049049180000000003, 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4099999999999999e-07 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403098+0000", 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7464000000000003e-05 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403106+0000", 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6629999999999999e-06 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403111+0000", 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403137+0000", 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6349000000000001e-05 2024-09-21T23:25:44.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403148+0000", 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.077e-05 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407920+0000", 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047716590000000001 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407935+0000", 2024-09-21T23:25:44.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4783999999999999e-05 2024-09-21T23:25:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.411 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.411 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:44.413 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:44.413+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:44.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:25:44.514 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:44.514+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:44.522 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:44.522 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:44.522 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:44.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:25:44.614 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:44.614+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:44.638 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:44.638 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:44.638 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:44.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:25:44.731 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:44.731+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:44.794 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:44.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 15.623728323, 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003527526, 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170743+0000", 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.08e-07 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170750+0000", 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800000000000001e-06 2024-09-21T23:25:44.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170767+0000", 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6923999999999999e-05 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170788+0000", 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0446e-05 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170849+0000", 2024-09-21T23:25:44.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1409999999999996e-05 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170881+0000", 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2075e-05 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174218+0000", 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033371070000000002 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174270+0000", 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1375999999999997e-05 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 14.607738829000001, 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011783570000000001, 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:44.799 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:44.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7999999999999996e-07 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186750+0000", 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7444000000000001e-05 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186834+0000", 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3974999999999995e-05 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:44.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186876+0000", 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2796000000000002e-05 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186999+0000", 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012303800000000001 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187036+0000", 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6897000000000003e-05 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187869+0000", 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00083215199999999996 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187910+0000", 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1575000000000001e-05 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:44.803 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:44.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:25:44.845 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:44.845+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:44.918 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:44.917+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:44.951 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:44.951 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:44.951 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:44.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:25:45.044 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:45.044+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:45.107 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:45.108 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:45.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:25:45.157 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:45.157+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:45.220 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:45.219+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:45.264 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:1 2.3 e19/18 2:c4c92e5a:::rbd_trash:head v 19'1, mlcod=0'0)", 2024-09-21T23:25:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.097182655000001, 2024-09-21T23:25:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039243489999999997, 2024-09-21T23:25:45.265 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:45.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2999999999999999e-07 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167667+0000", 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7169999999999997e-06 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:45.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167679+0000", 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.276e-05 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167688+0000", 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8380000000000008e-06 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167739+0000", 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0964999999999997e-05 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:45.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167762+0000", 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2221999999999998e-05 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171515+0000", 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037531119999999999 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171582+0000", 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7504999999999998e-05 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:45.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:45.270 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:45.270 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:45.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:25:45.321 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:45.321+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:45.421 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:45.421 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:45.421 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:45.421 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:45.422 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:45.421+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:45.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:25:45.522 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:45.522+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:45.536 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:45.536 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:45.537 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:45.537 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:45.537 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:45.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:25:45.623 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:45.623+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:45.692 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [] 2024-09-21T23:25:45.692 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:45.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:25:45.745 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:45.745+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:45.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T23:25:45.822 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:45.822 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:45.822 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:25:45.822 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:45.824 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:45.824+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:45.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:25:45.925 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:45.925+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:45.980 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:45.980 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:45.980 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:45.980 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:45.980 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:45.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:25:46.034 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:46.034+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:46.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:25:46.099 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:46.099 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:25:46.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:25:46.099 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-09-21T23:25:46.099 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:46.107 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:25:46.132 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:46.132+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:46.212 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:25:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:25:46.228 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:46.227+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:46.229 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:25:46.229 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:25:46.246 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:25:46.246 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:25:46.250 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-09-21T23:25:46.257 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:46.257 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:46.257 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:25:46.257 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:46.264 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:25:46.264 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:25:46.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:25:46.281 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:25:46.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:25:46.299 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:25:46.315 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:25:46.316 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:25:46.329 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:46.328+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:46.332 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:25:46.332 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:25:46.373 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:46.373 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:46.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:25:46.430 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:46.430+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:46.489 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:46.489 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:46.497 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:25:46.531 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:46.531+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:46.631 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:46.631+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:46.646 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:46.646 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:46.655 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:25:46.732 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:46.732+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:46.761 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:46.762 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:46.770 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:25:46.834 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:46.833+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:46.877 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:46.877 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:46.886 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:25:46.934 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:46.934+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:46.992 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:46.993 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:47.001 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:25:47.042 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:47.042+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:47.136 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:47.136+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:47.149 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:47.149 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:47.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:25:47.237 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:47.236+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:47.306 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:47.306 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:47.314 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:25:47.358 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:47.358+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:47.434 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:47.434 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:47.438 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:47.438+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:47.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:25:47.539 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:47.539+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:47.593 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:47.593 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:47.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:25:47.640 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:47.640+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:47.709 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:47.709 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:47.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:25:47.763 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:47.763+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:47.828 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:47.828 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:47.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:25:47.842 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:47.842+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:47.943 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:47.943+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:47.983 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:47.983 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:47.992 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:25:48.044 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:48.044+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:48.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:25:48.151 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:48.151+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:48.214 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:48.215 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:48.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:25:48.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T23:25:48.265 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:48.264+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:48.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:25:48.347 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:48.347+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:48.374 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:48.374 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:48.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:25:48.468 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:48.468+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:48.472 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:25:48.472 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:25:48.490 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:25:48.490 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:25:48.506 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:25:48.507 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:25:48.523 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:25:48.523 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:25:48.549 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:48.548+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:48.549 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:48.549 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:48.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:1 2.3 e19/18 2:c4c92e5a:::rbd_trash:head v 19'1, mlcod=0'0)", 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 20.381913150999999, 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039243489999999997, 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:48.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2999999999999999e-07 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:48.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167667+0000", 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7169999999999997e-06 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167679+0000", 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.276e-05 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167688+0000", 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8380000000000008e-06 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:48.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167739+0000", 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0964999999999997e-05 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167762+0000", 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2221999999999998e-05 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171515+0000", 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037531119999999999 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:48.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:48.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:48.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171582+0000", 2024-09-21T23:25:48.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7504999999999998e-05 2024-09-21T23:25:48.554 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:48.554 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:48.554 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:48.554 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:48.554 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:48.554 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:48.557 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:25:48.558 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:25:48.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:25:48.575 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:25:48.575 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:25:48.594 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:25:48.595 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:25:48.612 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:25:48.612 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:25:48.664 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:48.664+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:48.685 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:48.685 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:48.685 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:48.685 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:48.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:25:48.751 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:48.751+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:48.841 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:48.841 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:48.841 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:48.841 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:48.841 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:48.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:25:48.851 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:48.851+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:48.952 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:48.952+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:48.956 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:48.957 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:48.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:48.957 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [] 2024-09-21T23:25:48.957 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:48.966 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:25:49.053 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:49.053+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:49.086 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:49.086 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:49.086 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:25:49.086 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:49.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:25:49.154 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:49.154+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:49.203 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:49.203 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:49.204 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:49.204 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:49.204 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:49.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:25:49.246 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-09-21T23:25:49.255 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:49.255+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:49.256 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:25:49.256 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:25:49.257 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T23:25:49.257 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-09-21T23:25:49.257 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 0 2024-09-21T23:25:49.356 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:49.356+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:49.359 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:49.359 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:25:49.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:25:49.360 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-09-21T23:25:49.360 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:49.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:25:49.456 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:49.456+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:49.516 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:25:49.516 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:49.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:25:49.557 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:49.557+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:49.659 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:49.658+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:49.671 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:49.671 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:49.672 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:49.672 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:49.672 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:49.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:25:49.760 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:49.759+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:49.830 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:49.830 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:25:49.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:25:49.830 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:25:49.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.830 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:25:49.830 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 21.666191576999999, 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0080986310000000002, 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:49.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163709+0000", 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.832 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163727+0000", 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7552000000000001e-05 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163744+0000", 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7541999999999999e-05 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163757+0000", 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2855e-05 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164069+0000", 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031179899999999997 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164084+0000", 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5265e-05 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167127+0000", 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030427029999999999 2024-09-21T23:25:49.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167317+0000", 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000190006 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167465+0000", 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014824 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.168309+0000", 2024-09-21T23:25:49.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084350899999999999 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171730+0000", 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034217620000000001 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171770+0000", 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0074999999999998e-05 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:49.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171802+0000", 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1563999999999998e-05 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:1 2.3 e19/18 ondisk, result = 0)", 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 21.65824933, 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016281800000000001, 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6899999999999996e-07 2024-09-21T23:25:49.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171651+0000", 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0309999999999998e-06 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171665+0000", 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.309e-05 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:49.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171672+0000", 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6070000000000002e-06 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171716+0000", 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4344000000000001e-05 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171727+0000", 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0488e-05 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:49.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171808+0000", 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1489000000000005e-05 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 21.655473888, 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023803299999999999, 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:49.841 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 2 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174422+0000", 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4300000000000001e-06 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174429+0000", 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1659999999999999e-06 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174443+0000", 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4358e-05 2024-09-21T23:25:49.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174449+0000", 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2770000000000002e-06 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174540+0000", 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0209000000000003e-05 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174659+0000", 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011959300000000001 2024-09-21T23:25:49.844 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 21.655008888000001, 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016795100000000001, 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:49.845 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 3 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174887+0000", 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174892+0000", 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8180000000000003e-06 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174902+0000", 2024-09-21T23:25:49.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0095000000000001e-05 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174911+0000", 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0799999999999995e-06 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174974+0000", 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3027000000000005e-05 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.175054+0000", 2024-09-21T23:25:49.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0011000000000004e-05 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 20.661503257, 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006147263, 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:49.849 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168394+0000", 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.956e-06 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168404+0000", 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0463e-05 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168423+0000", 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8715999999999998e-05 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168432+0000", 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.516e-06 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168529+0000", 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7073999999999999e-05 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168534+0000", 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.831e-06 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170224+0000", 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016893590000000001 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170332+0000", 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108346 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170445+0000", 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112642 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.853 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.171210+0000", 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076543800000000001 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174469+0000", 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003259086 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174506+0000", 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6465999999999999e-05 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174539+0000", 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3370000000000001e-05 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:5 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 20.655561211999999, 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021298500000000001, 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:49.855 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174335+0000", 2024-09-21T23:25:49.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.111e-06 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174341+0000", 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6239999999999996e-06 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174356+0000", 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4548e-05 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:49.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174375+0000", 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8757e-05 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174457+0000", 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2702999999999997e-05 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174466+0000", 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.208e-06 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.858 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174547+0000", 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1033999999999997e-05 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.645789173000001, 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040438610000000002, 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:49.859 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:25:49.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184109+0000", 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9069999999999999e-06 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184124+0000", 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5780000000000001e-05 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:49.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184159+0000", 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4106999999999997e-05 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184178+0000", 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9908999999999999e-05 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184330+0000", 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015152300000000001 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:25:49.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184337+0000", 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7920000000000004e-06 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186239+0000", 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019020129999999999 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186307+0000", 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8326999999999998e-05 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186459+0000", 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152099 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187261+0000", 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00080186199999999997 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188085+0000", 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00082417199999999997 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.864 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188120+0000", 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5194999999999998e-05 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188150+0000", 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9175000000000001e-05 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:7 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:25:49.865 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.641918437000001, 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000179505, 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:49.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.16e-07 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187985+0000", 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3039999999999999e-06 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188000+0000", 2024-09-21T23:25:49.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5097e-05 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188007+0000", 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1330000000000001e-06 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188074+0000", 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7514000000000004e-05 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188082+0000", 2024-09-21T23:25:49.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3649999999999998e-06 2024-09-21T23:25:49.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:49.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:49.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:49.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188156+0000", 2024-09-21T23:25:49.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4176000000000003e-05 2024-09-21T23:25:49.869 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.869 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:49.869 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.869 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:49.869 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:49.869 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:49.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:25:49.871 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:49.870+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:49.971 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:49.970+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:50.018 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:25:50.018 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:50.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:25:50.072 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:50.072+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:50.134 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:50.134 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:50.134 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:50.135 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:50.135 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:50.144 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:25:50.185 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:50.185+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:50.254 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.0 primary-affinity to 0 (802) 2024-09-21T23:25:50.273 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:50.273+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:50.292 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:50.292 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:25:50.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:25:50.292 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-09-21T23:25:50.292 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:50.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:25:50.385 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:50.385+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:50.449 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:50.449 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:50.449 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:25:50.449 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:50.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:25:50.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:50.474+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:50.576 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:50.575+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:50.604 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:50.604 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:50.604 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:50.604 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:50.604 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:50.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:25:50.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:25:50.677 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:50.675+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:50.722 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:50.722 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:25:50.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.553752810999999, 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090066050000000009, 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 37 2024-09-21T23:25:50.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7099999999999998e-07 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167009+0000", 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.92e-06 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167013+0000", 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167017+0000", 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7280000000000001e-06 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170778+0000", 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037612399999999999 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170831+0000", 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2694000000000001e-05 2024-09-21T23:25:50.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170970+0000", 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000138966 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.171555+0000", 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058538399999999999 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175872+0000", 2024-09-21T23:25:50.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043172389999999996 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175969+0000", 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6844999999999999e-05 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.176012+0000", 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3356999999999998e-05 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.527647975000001, 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069023069999999999, 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.729 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0999999999999999e-07 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193115+0000", 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5700000000000001e-06 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193129+0000", 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4124000000000001e-05 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193133+0000", 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0550000000000001e-06 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194144+0000", 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001010735 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194231+0000", 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7665999999999997e-05 2024-09-21T23:25:50.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194357+0000", 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125353 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194844+0000", 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000487305 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199964+0000", 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051204859999999996 2024-09-21T23:25:50.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199992+0000", 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7902000000000002e-05 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.200013+0000", 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0601000000000001e-05 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.505998901000002, 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0067494620000000003, 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 73 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4900000000000001e-07 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214818+0000", 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7754000000000003e-05 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214842+0000", 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4552000000000001e-05 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214846+0000", 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9129999999999996e-06 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215813+0000", 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00096695899999999998 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215904+0000", 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0328000000000007e-05 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216020+0000", 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115838 2024-09-21T23:25:50.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217018+0000", 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099836200000000003 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221461+0000", 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044435969999999997 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221487+0000", 2024-09-21T23:25:50.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5806e-05 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221509+0000", 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1903999999999999e-05 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.505904198, 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068139230000000004, 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.740 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 74 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.741 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214855+0000", 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-07 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214859+0000", 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1389999999999997e-06 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214871+0000", 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1698999999999999e-05 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214887+0000", 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6061000000000001e-05 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216359+0000", 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014720779999999999 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216439+0000", 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0191999999999994e-05 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216579+0000", 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014029099999999999 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217129+0000", 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054932099999999997 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221559+0000", 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044299710000000004 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221627+0000", 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8631000000000001e-05 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221668+0000", 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0988000000000001e-05 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.489228012000002, 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0077112509999999997, 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.746 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 86 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.747 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5199999999999998e-07 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231533+0000", 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3879999999999998e-06 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231537+0000", 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0010000000000003e-06 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231540+0000", 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0489999999999999e-06 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233890+0000", 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023500249999999999 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233937+0000", 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6400999999999998e-05 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234015+0000", 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8343000000000005e-05 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235675+0000", 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001660124 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239138+0000", 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034624109999999999 2024-09-21T23:25:50.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239200+0000", 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2698000000000002e-05 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239242+0000", 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1458999999999997e-05 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.751 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.489211742999998, 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010200097999999999, 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.752 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2700000000000001e-07 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231588+0000", 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0985000000000001e-05 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231593+0000", 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1059999999999999e-06 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231606+0000", 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2343e-05 2024-09-21T23:25:50.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234406+0000", 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027999850000000001 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234489+0000", 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3051000000000002e-05 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234631+0000", 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285799999999999 2024-09-21T23:25:50.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235679+0000", 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001047517 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241644+0000", 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059652129999999996 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241704+0000", 2024-09-21T23:25:50.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9732999999999999e-05 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241747+0000", 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-05 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.489145904000001, 2024-09-21T23:25:50.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010246534, 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 88 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:50.758 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0199999999999998e-07 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231615+0000", 2024-09-21T23:25:50.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3439999999999999e-06 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231619+0000", 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0219999999999998e-06 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231622+0000", 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6989999999999999e-06 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235337+0000", 2024-09-21T23:25:50.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037150030000000001 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235385+0000", 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7861999999999997e-05 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235458+0000", 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2596999999999996e-05 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235978+0000", 2024-09-21T23:25:50.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052013600000000004 2024-09-21T23:25:50.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241799+0000", 2024-09-21T23:25:50.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058215289999999998 2024-09-21T23:25:50.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241836+0000", 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6282999999999998e-05 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241859+0000", 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3757e-05 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.763 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.331754720999999, 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072206969999999999, 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 112 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:50.764 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.72e-07 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389069+0000", 2024-09-21T23:25:50.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4875000000000005e-05 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389079+0000", 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0135e-05 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389085+0000", 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394006+0000", 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049207920000000002 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394052+0000", 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5810999999999999e-05 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394139+0000", 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7130000000000001e-05 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395407+0000", 2024-09-21T23:25:50.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012681260000000001 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396139+0000", 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000731703 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396194+0000", 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5309999999999997e-05 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396225+0000", 2024-09-21T23:25:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0589000000000001e-05 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.331647764, 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086923490000000003, 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.769 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 113 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389116+0000", 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.245e-06 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389126+0000", 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0295000000000001e-05 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389132+0000", 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0889999999999996e-06 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394548+0000", 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005415876 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394642+0000", 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4531999999999995e-05 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.772 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394714+0000", 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1753000000000002e-05 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396233+0000", 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015185299999999999 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397749+0000", 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015157770000000001 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397779+0000", 2024-09-21T23:25:50.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0561000000000001e-05 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397803+0000", 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4116000000000001e-05 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.331615695, 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.01139824, 2024-09-21T23:25:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 114 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.01e-07 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389146+0000", 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5239999999999999e-06 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.776 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389150+0000", 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3989999999999997e-06 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389153+0000", 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.038e-06 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395026+0000", 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058731720000000003 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.777 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395071+0000", 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4388999999999998e-05 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395141+0000", 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0053000000000002e-05 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397196+0000", 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020555899999999999 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.778 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400494+0000", 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032972679999999999 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400520+0000", 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6423000000000001e-05 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400541+0000", 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0982999999999999e-05 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.779 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.331597016, 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011537824, 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 115 2024-09-21T23:25:50.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4900000000000001e-07 2024-09-21T23:25:50.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389197+0000", 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5204999999999999e-05 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389203+0000", 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0979999999999999e-06 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389207+0000", 2024-09-21T23:25:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6600000000000001e-06 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395443+0000", 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062358099999999996 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395525+0000", 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1855999999999997e-05 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.783 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395669+0000", 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014472399999999999 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397204+0000", 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001534817 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400599+0000", 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033944499999999998 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.784 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400655+0000", 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6647000000000002e-05 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400699+0000", 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4208000000000001e-05 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.331545563999999, 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011571580999999999, 2024-09-21T23:25:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 116 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.786 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3099999999999999e-07 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389216+0000", 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4260000000000002e-06 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389223+0000", 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389225+0000", 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7760000000000002e-06 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396256+0000", 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070309020000000003 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396348+0000", 2024-09-21T23:25:50.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1780000000000006e-05 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396481+0000", 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000133101 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397208+0000", 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072716199999999995 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400737+0000", 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035290220000000001 2024-09-21T23:25:50.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400763+0000", 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5219999999999999e-05 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400785+0000", 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2045e-05 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.790 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.331528881000001, 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011687858000000001, 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 117 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.791 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3699999999999999e-07 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389232+0000", 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1789999999999998e-06 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389238+0000", 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209999999999999e-06 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389241+0000", 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.557e-06 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396792+0000", 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075506899999999997 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396838+0000", 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5639000000000001e-05 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396958+0000", 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119892 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397458+0000", 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049987600000000001 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400824+0000", 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033662560000000002 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400880+0000", 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5779999999999998e-05 2024-09-21T23:25:50.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400918+0000", 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8031000000000002e-05 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.319510406999999, 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006886924, 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.796 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 120 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.797 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.798 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:25:50.798 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.798 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.798 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401249+0000", 2024-09-21T23:25:50.798 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.13e-07 2024-09-21T23:25:50.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.798 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401286+0000", 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6773999999999997e-05 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401295+0000", 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9039999999999998e-06 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401299+0000", 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3250000000000001e-06 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402183+0000", 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000883758 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402264+0000", 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1080000000000003e-05 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402401+0000", 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013719700000000001 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403430+0000", 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010290119999999999 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408045+0000", 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046146659999999999 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408095+0000", 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0898e-05 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408135+0000", 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9697000000000003e-05 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 29.319451933, 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069210920000000002, 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.802 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 121 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.803 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3200000000000001e-07 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401341+0000", 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4236999999999998e-05 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401346+0000", 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.04e-06 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401350+0000", 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6399999999999999e-06 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402733+0000", 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013825790000000001 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.805 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402816+0000", 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3461000000000006e-05 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402941+0000", 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012518300000000001 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403533+0000", 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000592071 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.806 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408174+0000", 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046404209999999996 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408204+0000", 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0147000000000001e-05 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408228+0000", 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3981e-05 2024-09-21T23:25:50.807 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 28.883309036, 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064924400000000004, 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.808 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 139 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0800000000000001e-07 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837452+0000", 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7209999999999999e-06 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837455+0000", 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2279999999999998e-06 2024-09-21T23:25:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837458+0000", 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8200000000000001e-06 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842187+0000", 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047298460000000002 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842233+0000", 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6056000000000001e-05 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842304+0000", 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0862e-05 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843291+0000", 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00098632100000000007 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843894+0000", 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060369199999999997 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843922+0000", 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7429000000000001e-05 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843942+0000", 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0256999999999999e-05 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 28.883297092999999, 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066155550000000004, 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 140 2024-09-21T23:25:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1e-07 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837517+0000", 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5034000000000002e-05 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837522+0000", 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8520000000000003e-06 2024-09-21T23:25:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837525+0000", 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4139999999999998e-06 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842562+0000", 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050366279999999996 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842615+0000", 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3675999999999999e-05 2024-09-21T23:25:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842715+0000", 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9990999999999999e-05 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843873+0000", 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011573 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844026+0000", 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015368900000000001 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844058+0000", 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1449999999999999e-05 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844077+0000", 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9310999999999999e-05 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 28.883224899999998, 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068324850000000001, 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 141 2024-09-21T23:25:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.22e-07 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837539+0000", 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.023e-06 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837543+0000", 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6490000000000001e-06 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837546+0000", 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.382e-06 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843479+0000", 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059328189999999998 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843519+0000", 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0071000000000003e-05 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843600+0000", 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1376000000000002e-05 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844137+0000", 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053640299999999997 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844319+0000", 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000181893 2024-09-21T23:25:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844346+0000", 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.781e-05 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844366+0000", 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9737000000000001e-05 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:25:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 28.267891205000002, 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006594145, 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 158 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452868+0000", 2024-09-21T23:25:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9500000000000003e-07 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452938+0000", 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9598999999999996e-05 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452950+0000", 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2971e-05 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452960+0000", 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.448e-06 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456217+0000", 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003257378 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456557+0000", 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033989399999999998 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456703+0000", 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000145599 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459339+0000", 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026365529999999998 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459386+0000", 2024-09-21T23:25:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6640000000000001e-05 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459428+0000", 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2148999999999999e-05 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459462+0000", 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3419e-05 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 28.267786830999999, 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0065672760000000004, 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 159 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5400000000000001e-07 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452977+0000", 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.301e-06 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452984+0000", 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1840000000000002e-06 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452989+0000", 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5219999999999998e-06 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457324+0000", 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043348780000000003 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457416+0000", 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1824999999999996e-05 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457574+0000", 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015810999999999999 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459376+0000", 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001801649 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459467+0000", 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1367999999999997e-05 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459507+0000", 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9576000000000002e-05 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459539+0000", 2024-09-21T23:25:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2308999999999998e-05 2024-09-21T23:25:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:50.837 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:50.838 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:25:50.838 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:25:50.839 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:25:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:25:50.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:25:50.840 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:25:50.840 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:25:50.841 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:25:50.841 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:25:50.841 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:25:50.841 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:25:50.842 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:50.837+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:50.845 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:25:50.846 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:25:50.863 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:25:50.863 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:25:50.880 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:25:50.880 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:25:50.939 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:50.938+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:50.950 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:50.950 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:50.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:25:51.041 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:51.041+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:51.107 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:51.107 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:51.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:25:51.159 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:51.158+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:51.223 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:51.224 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:51.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:25:51.242 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:51.242+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:51.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T23:25:51.342 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:51.342+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:51.383 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:51.383 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:51.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:25:51.443 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:51.443+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:51.499 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:51.499 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:51.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:25:51.551 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:51.550+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:51.615 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:51.615 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:51.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:25:51.665 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:51.664+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:51.746 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:51.746+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:51.772 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:51.772 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:51.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:25:51.848 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:51.846+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:51.887 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:51.887 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:51.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:25:51.948 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:51.948+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:52.043 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:52.043 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:52.048 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:52.047+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:52.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:25:52.149 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:52.148+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:52.200 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:52.200 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:52.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:25:52.250 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:52.250+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:52.261 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-09-21T23:25:52.351 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:52.349+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:52.357 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:52.357 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:52.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:25:52.452 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:52.452+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:52.512 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:52.513 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:52.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:25:52.553 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:52.552+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:52.642 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:52.642 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:52.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:25:52.654 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:52.654+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:52.755 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:52.755+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:52.759 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:52.760 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:52.768 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:25:52.856 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:52.856+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:52.876 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:52.876 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:52.885 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:25:52.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:25:52.971 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:52.971+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:53.018 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:25:53.019 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:25:53.033 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:53.033 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:53.036 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:25:53.036 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:25:53.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:25:53.054 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:25:53.055 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:25:53.072 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:25:53.073 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:25:53.082 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:53.082+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:53.089 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:25:53.090 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:25:53.106 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:25:53.107 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:25:53.125 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:25:53.125 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:25:53.144 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:25:53.144 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:25:53.158 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:53.157+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:53.189 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:53.189 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:53.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:25:53.260 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:53.258+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:53.305 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:53.305 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:25:53.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.141452691000001, 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0080986310000000002, 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:25:53.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163709+0000", 2024-09-21T23:25:53.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163727+0000", 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7552000000000001e-05 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163744+0000", 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7541999999999999e-05 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163757+0000", 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2855e-05 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164069+0000", 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031179899999999997 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164084+0000", 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5265e-05 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167127+0000", 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030427029999999999 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167317+0000", 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000190006 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167465+0000", 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014824 2024-09-21T23:25:53.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:53.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.168309+0000", 2024-09-21T23:25:53.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084350899999999999 2024-09-21T23:25:53.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:53.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171730+0000", 2024-09-21T23:25:53.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034217620000000001 2024-09-21T23:25:53.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171770+0000", 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0074999999999998e-05 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171802+0000", 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1563999999999998e-05 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.312 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:1 2.3 e19/18 ondisk, result = 0)", 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.133510443999999, 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016281800000000001, 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6899999999999996e-07 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171651+0000", 2024-09-21T23:25:53.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0309999999999998e-06 2024-09-21T23:25:53.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:53.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171665+0000", 2024-09-21T23:25:53.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.309e-05 2024-09-21T23:25:53.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:53.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171672+0000", 2024-09-21T23:25:53.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6070000000000002e-06 2024-09-21T23:25:53.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171716+0000", 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4344000000000001e-05 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171727+0000", 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0488e-05 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171808+0000", 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1489000000000005e-05 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.130735002000002, 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023803299999999999, 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:53.317 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 2 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174422+0000", 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4300000000000001e-06 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174429+0000", 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1659999999999999e-06 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174443+0000", 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4358e-05 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174449+0000", 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2770000000000002e-06 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174540+0000", 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0209000000000003e-05 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174659+0000", 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011959300000000001 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.130270002, 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016795100000000001, 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 3 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174887+0000", 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174892+0000", 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8180000000000003e-06 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174902+0000", 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0095000000000001e-05 2024-09-21T23:25:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174911+0000", 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0799999999999995e-06 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174974+0000", 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3027000000000005e-05 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.175054+0000", 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0011000000000004e-05 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 24.136764371000002, 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006147263, 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168394+0000", 2024-09-21T23:25:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.956e-06 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168404+0000", 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0463e-05 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168423+0000", 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8715999999999998e-05 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168432+0000", 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.516e-06 2024-09-21T23:25:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168529+0000", 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7073999999999999e-05 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168534+0000", 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.831e-06 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170224+0000", 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016893590000000001 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170332+0000", 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108346 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170445+0000", 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112642 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.171210+0000", 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076543800000000001 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174469+0000", 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003259086 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174506+0000", 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6465999999999999e-05 2024-09-21T23:25:53.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:53.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174539+0000", 2024-09-21T23:25:53.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3370000000000001e-05 2024-09-21T23:25:53.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.331 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:53.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.332 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:5 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:25:53.332 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:25:53.332 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 24.130822326000001, 2024-09-21T23:25:53.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021298500000000001, 2024-09-21T23:25:53.332 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:53.332 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:53.332 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:25:53.332 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174335+0000", 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.111e-06 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:53.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174341+0000", 2024-09-21T23:25:53.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6239999999999996e-06 2024-09-21T23:25:53.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:53.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174356+0000", 2024-09-21T23:25:53.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4548e-05 2024-09-21T23:25:53.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:53.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174375+0000", 2024-09-21T23:25:53.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8757e-05 2024-09-21T23:25:53.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:53.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174457+0000", 2024-09-21T23:25:53.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2702999999999997e-05 2024-09-21T23:25:53.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:53.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174466+0000", 2024-09-21T23:25:53.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.208e-06 2024-09-21T23:25:53.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:53.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174547+0000", 2024-09-21T23:25:53.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1033999999999997e-05 2024-09-21T23:25:53.338 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.338 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:53.338 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.339 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-21T23:25:53.339 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:25:53.339 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 23.121050286999999, 2024-09-21T23:25:53.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040438610000000002, 2024-09-21T23:25:53.339 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:53.340 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:53.340 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:53.340 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:53.340 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:25:53.340 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:25:53.340 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7 2024-09-21T23:25:53.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.341 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:53.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:53.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:25:53.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:53.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:25:53.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:53.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184109+0000", 2024-09-21T23:25:53.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9069999999999999e-06 2024-09-21T23:25:53.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:53.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184124+0000", 2024-09-21T23:25:53.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5780000000000001e-05 2024-09-21T23:25:53.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:53.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184159+0000", 2024-09-21T23:25:53.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4106999999999997e-05 2024-09-21T23:25:53.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:53.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184178+0000", 2024-09-21T23:25:53.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9908999999999999e-05 2024-09-21T23:25:53.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:53.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184330+0000", 2024-09-21T23:25:53.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015152300000000001 2024-09-21T23:25:53.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:25:53.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184337+0000", 2024-09-21T23:25:53.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7920000000000004e-06 2024-09-21T23:25:53.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:53.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186239+0000", 2024-09-21T23:25:53.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019020129999999999 2024-09-21T23:25:53.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:53.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186307+0000", 2024-09-21T23:25:53.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8326999999999998e-05 2024-09-21T23:25:53.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:25:53.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186459+0000", 2024-09-21T23:25:53.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152099 2024-09-21T23:25:53.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187261+0000", 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00080186199999999997 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188085+0000", 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00082417199999999997 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:53.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188120+0000", 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5194999999999998e-05 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188150+0000", 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9175000000000001e-05 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:7 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 23.117179551, 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000179505, 2024-09-21T23:25:53.351 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:25:53.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.16e-07 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187985+0000", 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3039999999999999e-06 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188000+0000", 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5097e-05 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188007+0000", 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1330000000000001e-06 2024-09-21T23:25:53.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188074+0000", 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7514000000000004e-05 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188082+0000", 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3649999999999998e-06 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188156+0000", 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4176000000000003e-05 2024-09-21T23:25:53.354 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.355 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:53.355 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.355 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:53.355 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:53.355 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:53.356 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:25:53.360 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:53.359+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:53.460 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:53.459+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:53.463 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:53.463 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:53.463 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:25:53.463 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:53.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:25:53.562 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:53.562+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:53.578 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:53.578 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:53.578 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:53.578 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:53.578 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:53.587 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:25:53.671 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:53.671+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:53.734 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:53.735 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:25:53.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:25:53.735 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-09-21T23:25:53.735 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:53.743 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:25:53.784 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:53.784+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:53.864 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:53.863+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:53.890 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:53.890 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:53.890 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:25:53.890 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:53.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:25:53.965 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:53.965+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:54.005 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:54.005 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:54.006 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:54.006 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:54.006 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:54.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:25:54.097 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:54.097+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:54.163 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:54.163 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:25:54.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:25:54.163 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:25:54.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.163 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.163 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.994388078, 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090066050000000009, 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 37 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7099999999999998e-07 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167009+0000", 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.92e-06 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167013+0000", 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167017+0000", 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7280000000000001e-06 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170778+0000", 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037612399999999999 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170831+0000", 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2694000000000001e-05 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170970+0000", 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000138966 2024-09-21T23:25:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.171555+0000", 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058538399999999999 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175872+0000", 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043172389999999996 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175969+0000", 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6844999999999999e-05 2024-09-21T23:25:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.176012+0000", 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3356999999999998e-05 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.968283241999998, 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069023069999999999, 2024-09-21T23:25:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0999999999999999e-07 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193115+0000", 2024-09-21T23:25:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5700000000000001e-06 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193129+0000", 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4124000000000001e-05 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193133+0000", 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0550000000000001e-06 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194144+0000", 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001010735 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194231+0000", 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7665999999999997e-05 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194357+0000", 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125353 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194844+0000", 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000487305 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199964+0000", 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051204859999999996 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199992+0000", 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7902000000000002e-05 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.200013+0000", 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0601000000000001e-05 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.946634168000003, 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0067494620000000003, 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 73 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4900000000000001e-07 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214818+0000", 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7754000000000003e-05 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214842+0000", 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4552000000000001e-05 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214846+0000", 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9129999999999996e-06 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215813+0000", 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00096695899999999998 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215904+0000", 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0328000000000007e-05 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216020+0000", 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115838 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217018+0000", 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099836200000000003 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221461+0000", 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044435969999999997 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221487+0000", 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5806e-05 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221509+0000", 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1903999999999999e-05 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.946539465000001, 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068139230000000004, 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 74 2024-09-21T23:25:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214855+0000", 2024-09-21T23:25:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-07 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214859+0000", 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1389999999999997e-06 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214871+0000", 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1698999999999999e-05 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214887+0000", 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6061000000000001e-05 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216359+0000", 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014720779999999999 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216439+0000", 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0191999999999994e-05 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216579+0000", 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014029099999999999 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217129+0000", 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054932099999999997 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221559+0000", 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044299710000000004 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221627+0000", 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8631000000000001e-05 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221668+0000", 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0988000000000001e-05 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.929863279000003, 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0077112509999999997, 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 86 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5199999999999998e-07 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231533+0000", 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3879999999999998e-06 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231537+0000", 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0010000000000003e-06 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231540+0000", 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0489999999999999e-06 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233890+0000", 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023500249999999999 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233937+0000", 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6400999999999998e-05 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234015+0000", 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8343000000000005e-05 2024-09-21T23:25:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235675+0000", 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001660124 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239138+0000", 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034624109999999999 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239200+0000", 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2698000000000002e-05 2024-09-21T23:25:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239242+0000", 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1458999999999997e-05 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.929847010000003, 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010200097999999999, 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2700000000000001e-07 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231588+0000", 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0985000000000001e-05 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231593+0000", 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1059999999999999e-06 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231606+0000", 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2343e-05 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234406+0000", 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027999850000000001 2024-09-21T23:25:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234489+0000", 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3051000000000002e-05 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234631+0000", 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285799999999999 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235679+0000", 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001047517 2024-09-21T23:25:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241644+0000", 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059652129999999996 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241704+0000", 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9732999999999999e-05 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241747+0000", 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-05 2024-09-21T23:25:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.929781171000002, 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010246534, 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 88 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0199999999999998e-07 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231615+0000", 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3439999999999999e-06 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231619+0000", 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0219999999999998e-06 2024-09-21T23:25:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231622+0000", 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6989999999999999e-06 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235337+0000", 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037150030000000001 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235385+0000", 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7861999999999997e-05 2024-09-21T23:25:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235458+0000", 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2596999999999996e-05 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235978+0000", 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052013600000000004 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241799+0000", 2024-09-21T23:25:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058215289999999998 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241836+0000", 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6282999999999998e-05 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241859+0000", 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3757e-05 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.772389988, 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072206969999999999, 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 112 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.72e-07 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389069+0000", 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4875000000000005e-05 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389079+0000", 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0135e-05 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389085+0000", 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-09-21T23:25:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394006+0000", 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049207920000000002 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394052+0000", 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5810999999999999e-05 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394139+0000", 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7130000000000001e-05 2024-09-21T23:25:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395407+0000", 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012681260000000001 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396139+0000", 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000731703 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396194+0000", 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5309999999999997e-05 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396225+0000", 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0589000000000001e-05 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.772283031000001, 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086923490000000003, 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 113 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389116+0000", 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.245e-06 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389126+0000", 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0295000000000001e-05 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389132+0000", 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0889999999999996e-06 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394548+0000", 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005415876 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394642+0000", 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4531999999999995e-05 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394714+0000", 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1753000000000002e-05 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396233+0000", 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015185299999999999 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397749+0000", 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015157770000000001 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397779+0000", 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0561000000000001e-05 2024-09-21T23:25:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397803+0000", 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4116000000000001e-05 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.772250962000001, 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.01139824, 2024-09-21T23:25:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 114 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.01e-07 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389146+0000", 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5239999999999999e-06 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389150+0000", 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3989999999999997e-06 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389153+0000", 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.038e-06 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395026+0000", 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058731720000000003 2024-09-21T23:25:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395071+0000", 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4388999999999998e-05 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395141+0000", 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0053000000000002e-05 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397196+0000", 2024-09-21T23:25:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020555899999999999 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400494+0000", 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032972679999999999 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400520+0000", 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6423000000000001e-05 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400541+0000", 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0982999999999999e-05 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.772232283000001, 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011537824, 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 115 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4900000000000001e-07 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389197+0000", 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5204999999999999e-05 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389203+0000", 2024-09-21T23:25:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0979999999999999e-06 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389207+0000", 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6600000000000001e-06 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395443+0000", 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062358099999999996 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395525+0000", 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1855999999999997e-05 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395669+0000", 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014472399999999999 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397204+0000", 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001534817 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400599+0000", 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033944499999999998 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400655+0000", 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6647000000000002e-05 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400699+0000", 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4208000000000001e-05 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.772180831, 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011571580999999999, 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 116 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3099999999999999e-07 2024-09-21T23:25:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389216+0000", 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4260000000000002e-06 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389223+0000", 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389225+0000", 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7760000000000002e-06 2024-09-21T23:25:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396256+0000", 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070309020000000003 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396348+0000", 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1780000000000006e-05 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396481+0000", 2024-09-21T23:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000133101 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397208+0000", 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072716199999999995 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400737+0000", 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035290220000000001 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400763+0000", 2024-09-21T23:25:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5219999999999999e-05 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400785+0000", 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2045e-05 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.772164148000002, 2024-09-21T23:25:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011687858000000001, 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 117 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3699999999999999e-07 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389232+0000", 2024-09-21T23:25:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1789999999999998e-06 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389238+0000", 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209999999999999e-06 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389241+0000", 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.557e-06 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396792+0000", 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075506899999999997 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396838+0000", 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5639000000000001e-05 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396958+0000", 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119892 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397458+0000", 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049987600000000001 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400824+0000", 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033662560000000002 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400880+0000", 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5779999999999998e-05 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400918+0000", 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8031000000000002e-05 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.760145674, 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006886924, 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 120 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401249+0000", 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.13e-07 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401286+0000", 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6773999999999997e-05 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401295+0000", 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9039999999999998e-06 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401299+0000", 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3250000000000001e-06 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402183+0000", 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000883758 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402264+0000", 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1080000000000003e-05 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402401+0000", 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013719700000000001 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403430+0000", 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010290119999999999 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408045+0000", 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046146659999999999 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408095+0000", 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0898e-05 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408135+0000", 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9697000000000003e-05 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.760087200000001, 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069210920000000002, 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 121 2024-09-21T23:25:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3200000000000001e-07 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401341+0000", 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4236999999999998e-05 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401346+0000", 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.04e-06 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401350+0000", 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6399999999999999e-06 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402733+0000", 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013825790000000001 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402816+0000", 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3461000000000006e-05 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402941+0000", 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012518300000000001 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403533+0000", 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000592071 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408174+0000", 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046404209999999996 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408204+0000", 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0147000000000001e-05 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408228+0000", 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3981e-05 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.323944302999998, 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064924400000000004, 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 139 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0800000000000001e-07 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837452+0000", 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7209999999999999e-06 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837455+0000", 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2279999999999998e-06 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837458+0000", 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8200000000000001e-06 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842187+0000", 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047298460000000002 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842233+0000", 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6056000000000001e-05 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842304+0000", 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0862e-05 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843291+0000", 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00098632100000000007 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843894+0000", 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060369199999999997 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843922+0000", 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7429000000000001e-05 2024-09-21T23:25:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843942+0000", 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0256999999999999e-05 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.323932360000001, 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066155550000000004, 2024-09-21T23:25:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 140 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1e-07 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837517+0000", 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5034000000000002e-05 2024-09-21T23:25:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837522+0000", 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8520000000000003e-06 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837525+0000", 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4139999999999998e-06 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842562+0000", 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050366279999999996 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842615+0000", 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3675999999999999e-05 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842715+0000", 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9990999999999999e-05 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843873+0000", 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011573 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844026+0000", 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015368900000000001 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844058+0000", 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1449999999999999e-05 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844077+0000", 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9310999999999999e-05 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.323860166999999, 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068324850000000001, 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 141 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.22e-07 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837539+0000", 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.023e-06 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837543+0000", 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6490000000000001e-06 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837546+0000", 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.382e-06 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843479+0000", 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059328189999999998 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843519+0000", 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0071000000000003e-05 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843600+0000", 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1376000000000002e-05 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844137+0000", 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053640299999999997 2024-09-21T23:25:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844319+0000", 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000181893 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844346+0000", 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.781e-05 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844366+0000", 2024-09-21T23:25:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9737000000000001e-05 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.708526471999999, 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006594145, 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 158 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452868+0000", 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9500000000000003e-07 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452938+0000", 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9598999999999996e-05 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452950+0000", 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2971e-05 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452960+0000", 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.448e-06 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456217+0000", 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003257378 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456557+0000", 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033989399999999998 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456703+0000", 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000145599 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459339+0000", 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026365529999999998 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459386+0000", 2024-09-21T23:25:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6640000000000001e-05 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459428+0000", 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2148999999999999e-05 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459462+0000", 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3419e-05 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.708422098, 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0065672760000000004, 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 159 2024-09-21T23:25:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:25:54.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5400000000000001e-07 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452977+0000", 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.301e-06 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452984+0000", 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1840000000000002e-06 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452989+0000", 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5219999999999998e-06 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457324+0000", 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043348780000000003 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457416+0000", 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1824999999999996e-05 2024-09-21T23:25:54.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457574+0000", 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015810999999999999 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459376+0000", 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001801649 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459467+0000", 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1367999999999997e-05 2024-09-21T23:25:54.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459507+0000", 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9576000000000002e-05 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459539+0000", 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2308999999999998e-05 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:25:54.278 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:25:54.279 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:54.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T23:25:54.280 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:54.280+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:54.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:25:54.381 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:54.380+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:54.392 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:54.393 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:54.393 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:54.393 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:54.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:25:54.482 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:54.482+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:54.553 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:54.553 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:54.553 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:54.553 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:54.553 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:54.562 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:25:54.583 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:54.583+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:54.672 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.534902336000002, 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036127939999999999, 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:54.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3500000000000002e-07 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136608+0000", 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.182e-06 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136618+0000", 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9199999999999999e-06 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136628+0000", 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4390000000000006e-06 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136692+0000", 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3979000000000005e-05 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136716+0000", 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4391000000000001e-05 2024-09-21T23:25:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140171+0000", 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034554910000000002 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140214+0000", 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2057000000000001e-05 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'18)", 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.500384173, 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043957730000000004, 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171120+0000", 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6700000000000003e-07 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171125+0000", 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.451e-06 2024-09-21T23:25:54.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171138+0000", 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2541000000000001e-05 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171143+0000", 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4659999999999998e-06 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171187+0000", 2024-09-21T23:25:54.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4014000000000002e-05 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171200+0000", 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3108999999999999e-05 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175479+0000", 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042784570000000003 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175515+0000", 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5967999999999999e-05 2024-09-21T23:25:54.681 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:58 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.477631877, 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003032851, 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193872+0000", 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9400000000000004e-07 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193952+0000", 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9791999999999998e-05 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193962+0000", 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0034000000000001e-05 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193973+0000", 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0733e-05 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194015+0000", 2024-09-21T23:25:54.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2911999999999998e-05 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194031+0000", 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5082999999999999e-05 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196869+0000", 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002838808 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196904+0000", 2024-09-21T23:25:54.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4694999999999999e-05 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:59 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.477046211000001, 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053675709999999998, 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7100000000000002e-07 2024-09-21T23:25:54.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194462+0000", 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.668e-06 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194469+0000", 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1670000000000002e-06 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194475+0000", 2024-09-21T23:25:54.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-06 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194513+0000", 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7391999999999997e-05 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194531+0000", 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7870999999999999e-05 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199791+0000", 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052604119999999999 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199824+0000", 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3340000000000003e-05 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:60 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'38, mlcod=16'36)", 2024-09-21T23:25:54.690 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.470965516, 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003614562, 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:54.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200539+0000", 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5799999999999999e-06 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200551+0000", 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1816e-05 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200578+0000", 2024-09-21T23:25:54.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6772999999999999e-05 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200590+0000", 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2303000000000001e-05 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200715+0000", 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012523499999999999 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200731+0000", 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5690000000000001e-05 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204121+0000", 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003390363 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204152+0000", 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0802000000000001e-05 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:70 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'39)", 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.460909860999998, 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033896579999999998, 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:54.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210594+0000", 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1099999999999995e-07 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210600+0000", 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4389999999999997e-06 2024-09-21T23:25:54.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210613+0000", 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.243e-05 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210618+0000", 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0939999999999998e-06 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210843+0000", 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022493200000000001 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210858+0000", 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5109000000000001e-05 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213938+0000", 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030796500000000002 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213983+0000", 2024-09-21T23:25:54.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5293000000000002e-05 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:72 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'46, mlcod=16'43)", 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.455965085000003, 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029102279999999999, 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215539+0000", 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4199999999999995e-07 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215589+0000", 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0488999999999997e-05 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215598+0000", 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2830000000000007e-06 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215610+0000", 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1134999999999999e-05 2024-09-21T23:25:54.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215657+0000", 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7658999999999999e-05 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215679+0000", 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1526999999999999e-05 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218410+0000", 2024-09-21T23:25:54.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027316939999999998 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218448+0000", 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7799000000000002e-05 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:73 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'47, mlcod=16'43)", 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.455355738000002, 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051882020000000003, 2024-09-21T23:25:54.703 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216148+0000", 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7100000000000001e-07 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216218+0000", 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0313000000000003e-05 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216227+0000", 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9959999999999999e-06 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216232+0000", 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5340000000000001e-06 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216267+0000", 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4739000000000001e-05 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216281+0000", 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4345000000000001e-05 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221290+0000", 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050088989999999998 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221336+0000", 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5704999999999997e-05 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:74 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'48, mlcod=16'43)", 2024-09-21T23:25:54.707 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.454829795000002, 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046922939999999996, 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216674+0000", 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9900000000000001e-07 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216679+0000", 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.502e-06 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216687+0000", 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1790000000000008e-06 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216691+0000", 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7230000000000002e-06 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216737+0000", 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6087000000000001e-05 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216758+0000", 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0434000000000001e-05 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221338+0000", 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045804569999999996 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221366+0000", 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7413e-05 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:86 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'56, mlcod=16'50)", 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.437407860999997, 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049055269999999998, 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234096+0000", 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234099+0000", 2024-09-21T23:25:54.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3890000000000001e-06 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234107+0000", 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2349999999999999e-06 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234112+0000", 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0629999999999999e-06 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234143+0000", 2024-09-21T23:25:54.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1131999999999999e-05 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234155+0000", 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2140000000000001e-05 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.238972+0000", 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048167770000000004 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.239001+0000", 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9164000000000001e-05 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:87 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'57, mlcod=16'52)", 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.436728486, 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067179910000000004, 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.716 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0299999999999998e-07 2024-09-21T23:25:54.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234813+0000", 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7531999999999999e-05 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234826+0000", 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2617e-05 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234831+0000", 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1229999999999999e-06 2024-09-21T23:25:54.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234869+0000", 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8513999999999998e-05 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234881+0000", 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2148e-05 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241465+0000", 2024-09-21T23:25:54.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065837329999999996 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241493+0000", 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7521e-05 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:88 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.435930638000002, 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059344530000000001, 2024-09-21T23:25:54.720 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235573+0000", 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0100000000000004e-07 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235578+0000", 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1590000000000001e-06 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235590+0000", 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1229e-05 2024-09-21T23:25:54.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235595+0000", 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0889999999999999e-06 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235943+0000", 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000348227 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235957+0000", 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4419999999999999e-05 2024-09-21T23:25:54.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241494+0000", 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055371070000000003 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241507+0000", 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2520999999999999e-05 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:113 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'75, mlcod=16'71)", 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.276699360000002, 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028543489999999999, 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8199999999999999e-07 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394809+0000", 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-09-21T23:25:54.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394821+0000", 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2024999999999999e-05 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394827+0000", 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0020000000000003e-06 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394862+0000", 2024-09-21T23:25:54.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4811000000000003e-05 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394873+0000", 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.058e-05 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397631+0000", 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027581060000000002 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397658+0000", 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7073999999999999e-05 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'71)", 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.276287947, 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005160085, 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395216+0000", 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4999999999999998e-07 2024-09-21T23:25:54.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395221+0000", 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2290000000000002e-06 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395229+0000", 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2369999999999992e-06 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395233+0000", 2024-09-21T23:25:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1840000000000001e-06 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395260+0000", 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.728e-05 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395272+0000", 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1270000000000001e-05 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400341+0000", 2024-09-21T23:25:54.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050691809999999999 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400375+0000", 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4254000000000001e-05 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'72)", 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.275739426000001, 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046274580000000001, 2024-09-21T23:25:54.733 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395829+0000", 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4618999999999999e-05 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395836+0000", 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9090000000000003e-06 2024-09-21T23:25:54.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395839+0000", 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.371e-06 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395868+0000", 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9328000000000001e-05 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395880+0000", 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1501000000000001e-05 2024-09-21T23:25:54.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400376+0000", 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044966340000000002 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400391+0000", 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4676e-05 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.737 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:116 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'78, mlcod=16'73)", 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.274934879, 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038400970000000002, 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396569+0000", 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1100000000000001e-07 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396573+0000", 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5569999999999997e-06 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396579+0000", 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1120000000000001e-06 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396587+0000", 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396630+0000", 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2991000000000002e-05 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.740 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396642+0000", 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1481000000000001e-05 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400392+0000", 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037507830000000002 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400408+0000", 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5863e-05 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.741 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:117 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'79, mlcod=16'73)", 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.274432537000003, 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033512120000000001, 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7e-07 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397075+0000", 2024-09-21T23:25:54.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7809999999999999e-06 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397082+0000", 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397085+0000", 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7139999999999999e-06 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397130+0000", 2024-09-21T23:25:54.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.498e-05 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397142+0000", 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1838000000000001e-05 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400409+0000", 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032666349999999999 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400422+0000", 2024-09-21T23:25:54.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2866e-05 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.269608863999999, 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034761240000000001, 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.746 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401895+0000", 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7400000000000003e-07 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.747 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401963+0000", 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8701000000000005e-05 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401972+0000", 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9530000000000005e-06 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401977+0000", 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1579999999999998e-06 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402009+0000", 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2512000000000003e-05 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402022+0000", 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3220999999999999e-05 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405337+0000", 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033150300000000001 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405370+0000", 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2975000000000002e-05 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.269002794000002, 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054168369999999999, 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.750 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402501+0000", 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2699999999999999e-07 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402589+0000", 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899000000000007e-05 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402599+0000", 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0312000000000001e-05 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402605+0000", 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9689999999999999e-06 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402648+0000", 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3210000000000001e-05 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402664+0000", 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5347999999999999e-05 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.753 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407887+0000", 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052237919999999997 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407917+0000", 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9779999999999999e-05 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:54.754 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 33.268473413000002, 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049049180000000003, 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:54.755 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4099999999999999e-07 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403098+0000", 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7464000000000003e-05 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403106+0000", 2024-09-21T23:25:54.756 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6629999999999999e-06 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403111+0000", 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403137+0000", 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6349000000000001e-05 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403148+0000", 2024-09-21T23:25:54.757 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.077e-05 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407920+0000", 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047716590000000001 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407935+0000", 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4783999999999999e-05 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:54.758 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:54.759 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:54.760 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:25:54.760 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:54.758+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:54.861 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:54.861+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:54.914 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:54.914 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:54.914 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:54.914 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:54.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:25:54.967 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:54.966+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:55.031 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:55.031 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:55.031 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:55.032 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:55.032 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:55.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:25:55.062 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:55.062+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:55.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:25:55.194 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-21T23:25:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 26.023515256, 2024-09-21T23:25:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003527526, 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170743+0000", 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.08e-07 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170750+0000", 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800000000000001e-06 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170767+0000", 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6923999999999999e-05 2024-09-21T23:25:55.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170788+0000", 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0446e-05 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170849+0000", 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1409999999999996e-05 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170881+0000", 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2075e-05 2024-09-21T23:25:55.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174218+0000", 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033371070000000002 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174270+0000", 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1375999999999997e-05 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.198 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 25.007525762, 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011783570000000001, 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7999999999999996e-07 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186750+0000", 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7444000000000001e-05 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186834+0000", 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3974999999999995e-05 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186876+0000", 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2796000000000002e-05 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186999+0000", 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012303800000000001 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187036+0000", 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6897000000000003e-05 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187869+0000", 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00083215199999999996 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187910+0000", 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1575000000000001e-05 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.203 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.203 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:55.203 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:55.193+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:55.204 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:25:55.264 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:55.264+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:55.268 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-09-21T23:25:55.272 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:25:55.272 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:25:55.272 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T23:25:55.272 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T23:25:55.310 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:25:55.311 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:25:55.343 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:25:55.343 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:25:55.362 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:55.362 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:55.362 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:55.362 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:55.365 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:55.365+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:55.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:25:55.371 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:25:55.372 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:25:55.403 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:25:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:25:55.435 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:25:55.435 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:25:55.454 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:25:55.454 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:25:55.466 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:55.466+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:55.471 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:25:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:25:55.487 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:55.487 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:55.489 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:25:55.489 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:25:55.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:25:55.585 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:55.584+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:55.592 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:55.592 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:55.592 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 32, 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "d51cc561-ddb8-41ef-b61c-7c6d698e4dd0", 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-09-21T23:25:09.056809+0000", 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-09-21T23:25:55.263903+0000", 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-09-21T23:25:20.126996+0000", 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-09-21T23:25:11.226810+0000", 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5804032, 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "nodeep-scrub", 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-09-21T23:25:55.593 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 5, 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 2, 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 8, 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "jewel", 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-09-21T23:25:55.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:19.818212+0000", 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:25:55.595 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 1, 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 1, 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 1, 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 1, 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 1, 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:25:55.596 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "17", 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:25:55.597 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:25:55.598 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:25:55.599 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 6.9000000953674316, 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 6.9000000953674316, 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.28999999165534973, 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:25:55.600 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 2, 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "rbd", 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:26.968096+0000", 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 8193, 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:25:55.601 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 8, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 8, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 8, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 8, 2024-09-21T23:25:55.602 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 8, 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "21", 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:25:55.603 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 2, 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 21, 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:25:55.604 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:25:55.605 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:25:55.605 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:25:55.605 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:25:55.605 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:25:55.605 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:25:55.605 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:25:55.605 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:25:55.606 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:25:55.606 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:25:55.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.606 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:25:55.606 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:25:55.606 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:25:55.606 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:25:55.606 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:25:55.607 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:25:55.607 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:25:55.607 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:25:55.607 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:25:55.607 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:25:55.607 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:25:55.607 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-09-21T23:25:55.607 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:25:55.607 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd": {} 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 1.75, 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 1.75, 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 1, 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 1.75, 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 1.75, 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.608 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af", 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:55.609 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.612 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6817/2339966024", 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6819/2339966024", 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6823/2339966024", 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6821/2339966024", 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "88ee06a3-9a6a-41e8-9327-a7a61304c03d", 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:55.614 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.615 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.616 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.617 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6825/2889649822", 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6827/2889649822", 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6831/2889649822", 2024-09-21T23:25:55.619 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6829/2889649822", 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05", 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:55.620 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-09-21T23:25:55.621 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.622 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.623 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.624 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6809/864917651", 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6811/864917651", 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6815/864917651", 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6813/864917651", 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.625 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "562e8c30-0622-4cb3-b5d1-f15b42a444bc", 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6801/1713632981", 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6803/1713632981", 2024-09-21T23:25:55.630 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6807/1713632981", 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6805/1713632981", 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7030ff68-4556-4ad5-abc9-be758f1caa85", 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:55.631 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6808", 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:55.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6809", 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6810", 2024-09-21T23:25:55.633 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6811", 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6814", 2024-09-21T23:25:55.634 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6815", 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6812", 2024-09-21T23:25:55.635 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6813", 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6809/2660753894", 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6811/2660753894", 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6815/2660753894", 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6813/2660753894", 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:55.636 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7169a163-f067-4385-b734-f58a3ba9e89e", 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:25:55.637 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6824", 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6825", 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:55.638 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6826", 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6827", 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:55.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6830", 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6831", 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6828", 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6829", 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.641 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6825/2320687649", 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6827/2320687649", 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6831/2320687649", 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6829/2320687649", 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e511295d-2177-4dcb-a74b-b16b92534df6", 2024-09-21T23:25:55.642 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6800", 2024-09-21T23:25:55.643 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6801", 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6802", 2024-09-21T23:25:55.644 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6803", 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6806", 2024-09-21T23:25:55.645 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6807", 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.646 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6804", 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6805", 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6801/3490482162", 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6803/3490482162", 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3490482162", 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6805/3490482162", 2024-09-21T23:25:55.647 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "544f8a91-ce82-4855-85eb-ea814eb3d058", 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:25:55.648 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6816", 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6817", 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:55.649 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6818", 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6819", 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:55.650 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6822", 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6823", 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.651 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6820", 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6821", 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.652 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6817/1315554966", 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6819/1315554966", 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6823/1315554966", 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6821/1315554966", 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.514872+0000", 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:55.654 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.912129+0000", 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.745154+0000", 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:55.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:16.758871+0000", 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:55.656 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.287462+0000", 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:55.657 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:18.031636+0000", 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.055855+0000", 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:55.658 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.524609+0000", 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [], 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [], 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [], 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-09-21T23:25:55.659 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": {}, 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-09-21T23:25:55.660 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-09-21T23:25:55.661 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-09-21T23:25:55.661 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-09-21T23:25:55.661 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-09-21T23:25:55.661 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-09-21T23:25:55.661 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-09-21T23:25:55.661 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-09-21T23:25:55.661 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-09-21T23:25:55.661 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:55.661 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:55.663 INFO:tasks.thrashosds.thrasher:j is {'epoch': 32, 'fsid': 'd51cc561-ddb8-41ef-b61c-7c6d698e4dd0', 'created': '2024-09-21T23:25:09.056809+0000', 'modified': '2024-09-21T23:25:55.263903+0000', 'last_up_change': '2024-09-21T23:25:20.126996+0000', 'last_in_change': '2024-09-21T23:25:11.226810+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T23:25:19.818212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T23:25:26.968096+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 2339966024}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 2339966024}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 2339966024}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 2339966024}]}, 'public_addr': '172.21.15.31:6817/2339966024', 'cluster_addr': '172.21.15.31:6819/2339966024', 'heartbeat_back_addr': '172.21.15.31:6823/2339966024', 'heartbeat_front_addr': '172.21.15.31:6821/2339966024', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '88ee06a3-9a6a-41e8-9327-a7a61304c03d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 2889649822}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2889649822}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2889649822}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2889649822}]}, 'public_addr': '172.21.15.31:6825/2889649822', 'cluster_addr': '172.21.15.31:6827/2889649822', 'heartbeat_back_addr': '172.21.15.31:6831/2889649822', 'heartbeat_front_addr': '172.21.15.31:6829/2889649822', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 864917651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 864917651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 864917651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 864917651}]}, 'public_addr': '172.21.15.31:6809/864917651', 'cluster_addr': '172.21.15.31:6811/864917651', 'heartbeat_back_addr': '172.21.15.31:6815/864917651', 'heartbeat_front_addr': '172.21.15.31:6813/864917651', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '562e8c30-0622-4cb3-b5d1-f15b42a444bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6800', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6801', 'nonce': 1713632981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1713632981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1713632981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1713632981}]}, 'public_addr': '172.21.15.31:6801/1713632981', 'cluster_addr': '172.21.15.31:6803/1713632981', 'heartbeat_back_addr': '172.21.15.31:6807/1713632981', 'heartbeat_front_addr': '172.21.15.31:6805/1713632981', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7030ff68-4556-4ad5-abc9-be758f1caa85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6809', 'nonce': 2660753894}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6811', 'nonce': 2660753894}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6815', 'nonce': 2660753894}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6813', 'nonce': 2660753894}]}, 'public_addr': '172.21.15.60:6809/2660753894', 'cluster_addr': '172.21.15.60:6811/2660753894', 'heartbeat_back_addr': '172.21.15.60:6815/2660753894', 'heartbeat_front_addr': '172.21.15.60:6813/2660753894', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7169a163-f067-4385-b734-f58a3ba9e89e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6824', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6825', 'nonce': 2320687649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6826', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6827', 'nonce': 2320687649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6830', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6831', 'nonce': 2320687649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6828', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6829', 'nonce': 2320687649}]}, 'public_addr': '172.21.15.60:6825/2320687649', 'cluster_addr': '172.21.15.60:6827/2320687649', 'heartbeat_back_addr': '172.21.15.60:6831/2320687649', 'heartbeat_front_addr': '172.21.15.60:6829/2320687649', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e511295d-2177-4dcb-a74b-b16b92534df6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6801', 'nonce': 3490482162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6803', 'nonce': 3490482162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6807', 'nonce': 3490482162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6805', 'nonce': 3490482162}]}, 'public_addr': '172.21.15.60:6801/3490482162', 'cluster_addr': '172.21.15.60:6803/3490482162', 'heartbeat_back_addr': '172.21.15.60:6807/3490482162', 'heartbeat_front_addr': '172.21.15.60:6805/3490482162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '544f8a91-ce82-4855-85eb-ea814eb3d058', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6816', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6817', 'nonce': 1315554966}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6818', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6819', 'nonce': 1315554966}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6822', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6823', 'nonce': 1315554966}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6820', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6821', 'nonce': 1315554966}]}, 'public_addr': '172.21.15.60:6817/1315554966', 'cluster_addr': '172.21.15.60:6819/1315554966', 'heartbeat_back_addr': '172.21.15.60:6823/1315554966', 'heartbeat_front_addr': '172.21.15.60:6821/1315554966', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.514872+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.912129+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.745154+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:16.758871+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.287462+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:18.031636+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.055855+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.524609+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T23:25:55.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:25:55.664 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:55.664 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:55.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:25:55.667 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:55.667+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:55.769 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:55.769+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:55.815 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:55.815 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:55.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:25:55.869 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:55.869+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:55.941 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:55.941 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:55.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:25:55.995 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:55.994+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:56.033 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:25:56.033 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:25:56.051 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-21T23:25:55.780658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":235464,"kb_used_data":2592,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739256,"statfs":{"total":773094113280,"available":772852998144,"internally_reserved":0,"allocated":2654208,"data_stored":2131266,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001134"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-21T23:25:52.942275+0000","last_change":"2024-09-21T23:25:30.180125+0000","last_active":"2024-09-21T23:25:52.942275+0000","last_peered":"2024-09-21T23:25:52.942275+0000","last_clean":"2024-09-21T23:25:52.942275+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:25:52.942275+0000","last_undegraded":"2024-09-21T23:25:52.942275+0000","last_fullsized":"2024-09-21T23:25:52.942275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:17.515296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033759699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T23:25:51.062391+0000","last_change":"2024-09-21T23:25:30.189119+0000","last_active":"2024-09-21T23:25:51.062391+0000","last_peered":"2024-09-21T23:25:51.062391+0000","last_clean":"2024-09-21T23:25:51.062391+0000","last_became_active":"2024-09-21T23:25:28.165264+0000","last_became_peered":"2024-09-21T23:25:28.165264+0000","last_unstale":"2024-09-21T23:25:51.062391+0000","last_undegraded":"2024-09-21T23:25:51.062391+0000","last_fullsized":"2024-09-21T23:25:51.062391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:39:53.806934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053117699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T23:25:51.062798+0000","last_change":"2024-09-21T23:25:30.263565+0000","last_active":"2024-09-21T23:25:51.062798+0000","last_peered":"2024-09-21T23:25:51.062798+0000","last_clean":"2024-09-21T23:25:51.062798+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:25:51.062798+0000","last_undegraded":"2024-09-21T23:25:51.062798+0000","last_fullsized":"2024-09-21T23:25:51.062798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:46:37.545348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042085499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T23:25:51.062379+0000","last_change":"2024-09-21T23:25:30.189121+0000","last_active":"2024-09-21T23:25:51.062379+0000","last_peered":"2024-09-21T23:25:51.062379+0000","last_clean":"2024-09-21T23:25:51.062379+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:25:51.062379+0000","last_undegraded":"2024-09-21T23:25:51.062379+0000","last_fullsized":"2024-09-21T23:25:51.062379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:00:22.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054857199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T23:25:51.062604+0000","last_change":"2024-09-21T23:25:30.186984+0000","last_active":"2024-09-21T23:25:51.062604+0000","last_peered":"2024-09-21T23:25:51.062604+0000","last_clean":"2024-09-21T23:25:51.062604+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:25:51.062604+0000","last_undegraded":"2024-09-21T23:25:51.062604+0000","last_fullsized":"2024-09-21T23:25:51.062604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:01:38.740017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095847300000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T23:25:51.062903+0000","last_change":"2024-09-21T23:25:30.263343+0000","last_active":"2024-09-21T23:25:51.062903+0000","last_peered":"2024-09-21T23:25:51.062903+0000","last_clean":"2024-09-21T23:25:51.062903+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:25:51.062903+0000","last_undegraded":"2024-09-21T23:25:51.062903+0000","last_fullsized":"2024-09-21T23:25:51.062903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:50:16.699815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044148899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T23:25:51.062797+0000","last_change":"2024-09-21T23:25:30.263501+0000","last_active":"2024-09-21T23:25:51.062797+0000","last_peered":"2024-09-21T23:25:51.062797+0000","last_clean":"2024-09-21T23:25:51.062797+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:25:51.062797+0000","last_undegraded":"2024-09-21T23:25:51.062797+0000","last_fullsized":"2024-09-21T23:25:51.062797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.155893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036082099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T23:25:51.062607+0000","last_change":"2024-09-21T23:25:30.186477+0000","last_active":"2024-09-21T23:25:51.062607+0000","last_peered":"2024-09-21T23:25:51.062607+0000","last_clean":"2024-09-21T23:25:51.062607+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:25:51.062607+0000","last_undegraded":"2024-09-21T23:25:51.062607+0000","last_fullsized":"2024-09-21T23:25:51.062607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:32:47.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044681799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":188,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T23:25:51.062805+0000","last_change":"2024-09-21T23:25:21.135362+0000","last_active":"2024-09-21T23:25:51.062805+0000","last_peered":"2024-09-21T23:25:51.062805+0000","last_clean":"2024-09-21T23:25:51.062805+0000","last_became_active":"2024-09-21T23:25:21.135038+0000","last_became_peered":"2024-09-21T23:25:21.135038+0000","last_unstale":"2024-09-21T23:25:51.062805+0000","last_undegraded":"2024-09-21T23:25:51.062805+0000","last_fullsized":"2024-09-21T23:25:51.062805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:42:58.183351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":17484,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":364544,"data_stored":705094,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509451,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31492,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340348,"statfs":{"total":96636764160,"available":96604516352,"internally_reserved":0,"allocated":311296,"data_stored":114745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31504,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":319488,"data_stored":120176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509450,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":307200,"data_stored":114726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509451,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":311296,"data_stored":114745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":710544,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31512,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340328,"statfs":{"total":96636764160,"available":96604495872,"internally_reserved":0,"allocated":331776,"data_stored":125618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":331776,"data_stored":125618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:25:56.052 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [3, 4] 2024-09-21T23:25:56.052 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '3', '4'] 2024-09-21T23:25:56.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 1.0 3 4 2024-09-21T23:25:56.066 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:56.066 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:56.071 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:56.070+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:56.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:25:56.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:56.170+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:56.196 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:56.196 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:56.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:25:56.273 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:56.271+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:56.314 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:56.314 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:56.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:25:56.374 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:56.374+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:56.430 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:56.431 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:56.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:25:56.475 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:56.473+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:56.576 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:56.576+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:56.586 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:56.587 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:56.595 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:25:56.680 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:56.680+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:56.742 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:56.743 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:56.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:25:56.792 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:56.792+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:56.878 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:56.876+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:56.898 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:56.898 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:56.907 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:25:56.979 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:56.979+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:57.013 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:57.014 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:57.022 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:25:57.081 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:57.079+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:57.169 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:57.169 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:57.178 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:25:57.181 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:57.180+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:57.274 INFO:teuthology.orchestra.run.smithi031.stderr:set 1.0 pg_upmap mapping to [3,4] 2024-09-21T23:25:57.282 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:57.282+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:57.286 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:57.286 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:57.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T23:25:57.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:25:57.383 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:57.382+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:57.444 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:57.444 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:57.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:25:57.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:57.482+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:57.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:25:57.560 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:57.560 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:57.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:25:57.612 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:57.612+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:57.639 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:25:57.639 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:25:57.656 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:25:57.656 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:25:57.675 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:57.675 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:57.676 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:25:57.677 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:25:57.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:25:57.685 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:57.684+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:57.694 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:25:57.694 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:25:57.712 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:25:57.713 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:25:57.730 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:25:57.730 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:25:57.752 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:25:57.752 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:25:57.769 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:25:57.769 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:25:57.800 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:57.800 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:57.800 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:57.800 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:57.800 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:57.799+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:57.810 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:25:57.894 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:57.893+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:57.958 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:57.958 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:57.959 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:57.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:25:57.988 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:25:57.988+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:25:58.089 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:58.089+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:58.120 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.982918155999997, 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036127939999999999, 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3500000000000002e-07 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136608+0000", 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.182e-06 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136618+0000", 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9199999999999999e-06 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136628+0000", 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4390000000000006e-06 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136692+0000", 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3979000000000005e-05 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136716+0000", 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4391000000000001e-05 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140171+0000", 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034554910000000002 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140214+0000", 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2057000000000001e-05 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'18)", 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.948399993000002, 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043957730000000004, 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.125 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171120+0000", 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6700000000000003e-07 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171125+0000", 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.451e-06 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171138+0000", 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2541000000000001e-05 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171143+0000", 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4659999999999998e-06 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171187+0000", 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4014000000000002e-05 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171200+0000", 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3108999999999999e-05 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175479+0000", 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042784570000000003 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175515+0000", 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5967999999999999e-05 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:58 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:58.129 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.925647697000002, 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003032851, 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193872+0000", 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9400000000000004e-07 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193952+0000", 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9791999999999998e-05 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193962+0000", 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0034000000000001e-05 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193973+0000", 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0733e-05 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194015+0000", 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2911999999999998e-05 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194031+0000", 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5082999999999999e-05 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196869+0000", 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002838808 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196904+0000", 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4694999999999999e-05 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.133 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:59 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.925062031000003, 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053675709999999998, 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:58.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7100000000000002e-07 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194462+0000", 2024-09-21T23:25:58.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.668e-06 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194469+0000", 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1670000000000002e-06 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194475+0000", 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-06 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194513+0000", 2024-09-21T23:25:58.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7391999999999997e-05 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194531+0000", 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7870999999999999e-05 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199791+0000", 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052604119999999999 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199824+0000", 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3340000000000003e-05 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:60 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'38, mlcod=16'36)", 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.918981336000002, 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003614562, 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.138 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200539+0000", 2024-09-21T23:25:58.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5799999999999999e-06 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200551+0000", 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1816e-05 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200578+0000", 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6772999999999999e-05 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200590+0000", 2024-09-21T23:25:58.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2303000000000001e-05 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200715+0000", 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012523499999999999 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200731+0000", 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5690000000000001e-05 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204121+0000", 2024-09-21T23:25:58.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003390363 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204152+0000", 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0802000000000001e-05 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:70 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'39)", 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.908925680999999, 2024-09-21T23:25:58.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033896579999999998, 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210594+0000", 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1099999999999995e-07 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210600+0000", 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4389999999999997e-06 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210613+0000", 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.243e-05 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210618+0000", 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0939999999999998e-06 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210843+0000", 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022493200000000001 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210858+0000", 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5109000000000001e-05 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213938+0000", 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030796500000000002 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213983+0000", 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5293000000000002e-05 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:72 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'46, mlcod=16'43)", 2024-09-21T23:25:58.146 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.903980904999997, 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029102279999999999, 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215539+0000", 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4199999999999995e-07 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215589+0000", 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0488999999999997e-05 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215598+0000", 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2830000000000007e-06 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215610+0000", 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1134999999999999e-05 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215657+0000", 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7658999999999999e-05 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215679+0000", 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1526999999999999e-05 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218410+0000", 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027316939999999998 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218448+0000", 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7799000000000002e-05 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:73 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'47, mlcod=16'43)", 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.903371558000003, 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051882020000000003, 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:58.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216148+0000", 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7100000000000001e-07 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216218+0000", 2024-09-21T23:25:58.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0313000000000003e-05 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216227+0000", 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9959999999999999e-06 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216232+0000", 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5340000000000001e-06 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216267+0000", 2024-09-21T23:25:58.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4739000000000001e-05 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216281+0000", 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4345000000000001e-05 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221290+0000", 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050088989999999998 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221336+0000", 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5704999999999997e-05 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:74 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'48, mlcod=16'43)", 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.902845614999997, 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046922939999999996, 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216674+0000", 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9900000000000001e-07 2024-09-21T23:25:58.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216679+0000", 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.502e-06 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216687+0000", 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1790000000000008e-06 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216691+0000", 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7230000000000002e-06 2024-09-21T23:25:58.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216737+0000", 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6087000000000001e-05 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216758+0000", 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0434000000000001e-05 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221338+0000", 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045804569999999996 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221366+0000", 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7413e-05 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:86 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'56, mlcod=16'50)", 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.885423680999999, 2024-09-21T23:25:58.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049055269999999998, 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234096+0000", 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234099+0000", 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3890000000000001e-06 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234107+0000", 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2349999999999999e-06 2024-09-21T23:25:58.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234112+0000", 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0629999999999999e-06 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234143+0000", 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1131999999999999e-05 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234155+0000", 2024-09-21T23:25:58.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2140000000000001e-05 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.238972+0000", 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048167770000000004 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.239001+0000", 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9164000000000001e-05 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:87 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'57, mlcod=16'52)", 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.884744306000002, 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067179910000000004, 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0299999999999998e-07 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234813+0000", 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7531999999999999e-05 2024-09-21T23:25:58.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234826+0000", 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2617e-05 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234831+0000", 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1229999999999999e-06 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234869+0000", 2024-09-21T23:25:58.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8513999999999998e-05 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234881+0000", 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2148e-05 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241465+0000", 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065837329999999996 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241493+0000", 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7521e-05 2024-09-21T23:25:58.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:88 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.883946457999997, 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059344530000000001, 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235573+0000", 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0100000000000004e-07 2024-09-21T23:25:58.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235578+0000", 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1590000000000001e-06 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235590+0000", 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1229e-05 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235595+0000", 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0889999999999999e-06 2024-09-21T23:25:58.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235943+0000", 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000348227 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235957+0000", 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4419999999999999e-05 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241494+0000", 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055371070000000003 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241507+0000", 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2520999999999999e-05 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:113 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'75, mlcod=16'71)", 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.724715179999997, 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028543489999999999, 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.172 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8199999999999999e-07 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394809+0000", 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394821+0000", 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2024999999999999e-05 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394827+0000", 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0020000000000003e-06 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394862+0000", 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4811000000000003e-05 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394873+0000", 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.058e-05 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397631+0000", 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027581060000000002 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397658+0000", 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7073999999999999e-05 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'71)", 2024-09-21T23:25:58.176 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.724303767000002, 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005160085, 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395216+0000", 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4999999999999998e-07 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395221+0000", 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2290000000000002e-06 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395229+0000", 2024-09-21T23:25:58.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2369999999999992e-06 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395233+0000", 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1840000000000001e-06 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395260+0000", 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.728e-05 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395272+0000", 2024-09-21T23:25:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1270000000000001e-05 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400341+0000", 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050691809999999999 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400375+0000", 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4254000000000001e-05 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'72)", 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.723755246000003, 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046274580000000001, 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395829+0000", 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4618999999999999e-05 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395836+0000", 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9090000000000003e-06 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395839+0000", 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.371e-06 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395868+0000", 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9328000000000001e-05 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395880+0000", 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1501000000000001e-05 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400376+0000", 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044966340000000002 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400391+0000", 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4676e-05 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.184 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:116 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'78, mlcod=16'73)", 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.722950699000002, 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038400970000000002, 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396569+0000", 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1100000000000001e-07 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396573+0000", 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5569999999999997e-06 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396579+0000", 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1120000000000001e-06 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396587+0000", 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396630+0000", 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2991000000000002e-05 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396642+0000", 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1481000000000001e-05 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400392+0000", 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037507830000000002 2024-09-21T23:25:58.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400408+0000", 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5863e-05 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:117 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'79, mlcod=16'73)", 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.722448356999998, 2024-09-21T23:25:58.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033512120000000001, 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7e-07 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397075+0000", 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7809999999999999e-06 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397082+0000", 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397085+0000", 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7139999999999999e-06 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397130+0000", 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.498e-05 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397142+0000", 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1838000000000001e-05 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400409+0000", 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032666349999999999 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400422+0000", 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2866e-05 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.193 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.717624684, 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034761240000000001, 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401895+0000", 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7400000000000003e-07 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401963+0000", 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8701000000000005e-05 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401972+0000", 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9530000000000005e-06 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401977+0000", 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1579999999999998e-06 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402009+0000", 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2512000000000003e-05 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402022+0000", 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3220999999999999e-05 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405337+0000", 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033150300000000001 2024-09-21T23:25:58.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405370+0000", 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2975000000000002e-05 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.717018613999997, 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054168369999999999, 2024-09-21T23:25:58.198 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402501+0000", 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2699999999999999e-07 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402589+0000", 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899000000000007e-05 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402599+0000", 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0312000000000001e-05 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402605+0000", 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9689999999999999e-06 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402648+0000", 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3210000000000001e-05 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402664+0000", 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5347999999999999e-05 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407887+0000", 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052237919999999997 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407917+0000", 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9779999999999999e-05 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:58.202 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.716489232999997, 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049049180000000003, 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:58.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4099999999999999e-07 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403098+0000", 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7464000000000003e-05 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403106+0000", 2024-09-21T23:25:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6629999999999999e-06 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403111+0000", 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403137+0000", 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6349000000000001e-05 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403148+0000", 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.077e-05 2024-09-21T23:25:58.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407920+0000", 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047716590000000001 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407935+0000", 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4783999999999999e-05 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.206 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.207 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:58.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:25:58.208 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:58.208+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-09-21T23:25:58.308 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:58.307+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:58.330 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:58.330 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:58.330 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:58.330 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:58.339 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:25:58.410 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:25:58.410+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:25:58.490 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:58.490 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:58.490 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:58.490 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:58.490 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:58.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:25:58.543 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:58.542+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:58.609 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:58.609 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:58.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:58.609 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 29.439087998000002, 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003527526, 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170743+0000", 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.08e-07 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170750+0000", 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800000000000001e-06 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170767+0000", 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6923999999999999e-05 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170788+0000", 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0446e-05 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170849+0000", 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1409999999999996e-05 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170881+0000", 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2075e-05 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174218+0000", 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033371070000000002 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174270+0000", 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1375999999999997e-05 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 28.423098503999999, 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011783570000000001, 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:58.614 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:58.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7999999999999996e-07 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186750+0000", 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7444000000000001e-05 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186834+0000", 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3974999999999995e-05 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186876+0000", 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2796000000000002e-05 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186999+0000", 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012303800000000001 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187036+0000", 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6897000000000003e-05 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187869+0000", 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00083215199999999996 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187910+0000", 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1575000000000001e-05 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:58.618 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:58.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:25:58.620 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:58.618+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:58.720 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:58.719+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:58.774 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:58.774 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:58.775 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:58.775 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:58.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:25:58.821 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:25:58.821+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:25:58.922 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:58.920+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:58.944 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:58.945 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:58.945 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:58.945 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:58.945 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:58.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:25:59.040 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:25:59.039+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:25:59.107 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:59.108 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:59.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:59.108 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:59.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.108 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:1 2.3 e19/18 2:c4c92e5a:::rbd_trash:head v 19'1, mlcod=0'0)", 2024-09-21T23:25:59.108 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:59.108 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 30.940265692000001, 2024-09-21T23:25:59.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039243489999999997, 2024-09-21T23:25:59.108 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:59.108 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:59.108 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:59.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2999999999999999e-07 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167667+0000", 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7169999999999997e-06 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167679+0000", 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.276e-05 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.110 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167688+0000", 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8380000000000008e-06 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167739+0000", 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0964999999999997e-05 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167762+0000", 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2221999999999998e-05 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171515+0000", 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037531119999999999 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171582+0000", 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7504999999999998e-05 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:59.112 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:59.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:25:59.158 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:59.157+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:59.224 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:59.223+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:59.266 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:59.266 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:59.266 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:25:59.267 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:59.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:25:59.325 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:59.325+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:59.423 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:59.423 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:25:59.423 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:25:59.423 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:25:59.423 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:59.425 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:25:59.424+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:25:59.432 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:25:59.526 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:59.526+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:59.580 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.27911854200000003, 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00081488299999999997, 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:25:59.581 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301851+0000", 2024-09-21T23:25:59.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7899999999999998e-07 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302047+0000", 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019639200000000001 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302064+0000", 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7380999999999998e-05 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:59.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302080+0000", 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5237e-05 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302153+0000", 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3922000000000001e-05 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302162+0000", 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8170000000000005e-06 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302665+0000", 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050245499999999996 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.27633532399999999, 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00068581899999999995, 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:25:59.585 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:25:59.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304634+0000", 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5499999999999998e-07 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304796+0000", 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016237700000000001 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304813+0000", 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.732e-05 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304825+0000", 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1302e-05 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304896+0000", 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1729000000000004e-05 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304905+0000", 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8359999999999998e-06 2024-09-21T23:25:59.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:25:59.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:25:59.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:25:59.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.305319+0000", 2024-09-21T23:25:59.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00041360000000000002 2024-09-21T23:25:59.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:59.589 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:59.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:59.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:25:59.589 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:25:59.589 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:25:59.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:25:59.634 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:59.634+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:59.711 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:25:59.711 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:59.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:25:59.761 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:25:59.761+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:25:59.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:25:59.828 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:25:59.827+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:25:59.869 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:25:59.869 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:25:59.869 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:25:59.869 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:25:59.869 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:25:59.878 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:25:59.914 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:25:59.915 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:25:59.929 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:25:59.927+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:25:59.933 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:25:59.933 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:25:59.950 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:25:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:25:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:25:59.966 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:25:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:25:59.985 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:25:59.986 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:25:59.986 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:25:59.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:00.004 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:00.024 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:00.030 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:00.028+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:00.041 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:00.101 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:00.102 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:00.110 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:00.131 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:00.130+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:00.217 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:00.217 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:00.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:00.232 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:00.230+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:00.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T23:26:00.333 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:00.333+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:00.334 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:00.334 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:00.343 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:00.433 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:00.433+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:00.491 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:00.491 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:00.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:00.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:00.533+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:00.635 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:00.635+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:00.647 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:00.647 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:00.656 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:00.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:00.735+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:00.804 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:00.804 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:00.812 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:00.856 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:00.856+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:00.919 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:00.919 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:00.928 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:00.969 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:00.969+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:01.038 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:01.038+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:01.075 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:01.075 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:01.084 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:01.138 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:01.137+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:01.231 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:01.232 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:01.239 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:01.238+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:01.241 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:01.308 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-09-21T23:26:01.340 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:01.339+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:01.360 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:01.361 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:01.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:01.455 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:01.454+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:01.521 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:01.521 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:01.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:01.571 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:01.570+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:01.643 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:01.643+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:01.684 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:01.685 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:01.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:01.744 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:01.742+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:01.806 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:01.806 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:01.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:01.845 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:01.845+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:01.945 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:01.944+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:01.975 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:01.975 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:01.985 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:02.046 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:02.046+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:02.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:26:02.137 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:02.137 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:02.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:02.147 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:02.145+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:02.200 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:02.200 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:02.219 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:02.220 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:02.237 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:02.237 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:02.247 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:02.246+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:02.257 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:02.257 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:02.278 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:02.279 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:02.291 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:26:02.291 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:26:02.291 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T23:26:02.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T23:26:02.297 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:02.298 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:02.298 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:02.298 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:02.299 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:02.299 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:02.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:02.319 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:02.319 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:02.342 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:02.342 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:02.348 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:02.347+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:02.447 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:02.447 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:02.447 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:02.447 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:02.447 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:02.448 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:02.447+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:02.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:02.550 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:02.550+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:02.600 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:26:02.600 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:02.600 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 37, 2024-09-21T23:26:02.600 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "d51cc561-ddb8-41ef-b61c-7c6d698e4dd0", 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-09-21T23:25:09.056809+0000", 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-09-21T23:26:01.307042+0000", 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-09-21T23:25:20.126996+0000", 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-09-21T23:25:11.226810+0000", 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5801984, 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "noscrub", 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 5, 2024-09-21T23:26:02.601 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 2, 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 8, 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "luminous", 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-21T23:26:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:19.818212+0000", 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:02.603 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 1, 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 1, 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 1, 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 1, 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 1, 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "17", 2024-09-21T23:26:02.604 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:02.605 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:02.606 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:02.606 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:02.606 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:02.606 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:02.606 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:02.606 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:02.606 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:02.606 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:02.606 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:02.606 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:02.606 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:02.607 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:02.608 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:02.608 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-09-21T23:26:02.608 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-09-21T23:26:02.608 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-09-21T23:26:02.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.608 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:02.608 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-09-21T23:26:02.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.608 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:02.608 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 6.9000000953674316, 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 6.9000000953674316, 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.28999999165534973, 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 2, 2024-09-21T23:26:02.609 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "rbd", 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:26.968096+0000", 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 8193, 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:02.610 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 8, 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 8, 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 8, 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 8, 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 8, 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.611 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "21", 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 2, 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 21, 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:02.612 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:02.613 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd": {} 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:02.614 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 1.75, 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 1.75, 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 1, 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 1.75, 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 1.75, 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:02.615 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af", 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 35, 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.616 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-09-21T23:26:02.617 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:02.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.618 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-09-21T23:26:02.618 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:02.618 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.618 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.618 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:02.618 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:02.619 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6817/2339966024", 2024-09-21T23:26:02.620 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6819/2339966024", 2024-09-21T23:26:02.621 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6823/2339966024", 2024-09-21T23:26:02.621 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6821/2339966024", 2024-09-21T23:26:02.621 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:02.621 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:02.621 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:02.621 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.621 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:02.621 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "88ee06a3-9a6a-41e8-9327-a7a61304c03d", 2024-09-21T23:26:02.621 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.622 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-09-21T23:26:02.623 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-09-21T23:26:02.624 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-09-21T23:26:02.625 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6825/2889649822", 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6827/2889649822", 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6831/2889649822", 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6829/2889649822", 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05", 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-09-21T23:26:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-09-21T23:26:02.631 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6809/864917651", 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6811/864917651", 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6815/864917651", 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6813/864917651", 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "562e8c30-0622-4cb3-b5d1-f15b42a444bc", 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 36, 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.634 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-09-21T23:26:02.635 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:02.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6801/1713632981", 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6803/1713632981", 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6807/1713632981", 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6805/1713632981", 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:02.637 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7030ff68-4556-4ad5-abc9-be758f1caa85", 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:02.638 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:02.639 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:02.639 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.639 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.639 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6808", 2024-09-21T23:26:02.639 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:02.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.639 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.639 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6809", 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6810", 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6811", 2024-09-21T23:26:02.640 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6814", 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6815", 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:02.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6812", 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6813", 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:02.642 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6809/2660753894", 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6811/2660753894", 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6815/2660753894", 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6813/2660753894", 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7169a163-f067-4385-b734-f58a3ba9e89e", 2024-09-21T23:26:02.643 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.644 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6824", 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6825", 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6826", 2024-09-21T23:26:02.645 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6827", 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6830", 2024-09-21T23:26:02.646 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6831", 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6828", 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:02.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6829", 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6825/2320687649", 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6827/2320687649", 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6831/2320687649", 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6829/2320687649", 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:02.648 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.649 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:02.649 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e511295d-2177-4dcb-a74b-b16b92534df6", 2024-09-21T23:26:02.649 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:02.649 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:02.649 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:02.649 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:02.649 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:02.649 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6800", 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6801", 2024-09-21T23:26:02.650 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6802", 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6803", 2024-09-21T23:26:02.651 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6806", 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6807", 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:02.652 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6804", 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6805", 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.653 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6801/3490482162", 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6803/3490482162", 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3490482162", 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6805/3490482162", 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "544f8a91-ce82-4855-85eb-ea814eb3d058", 2024-09-21T23:26:02.654 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6816", 2024-09-21T23:26:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6817", 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6818", 2024-09-21T23:26:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6819", 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6822", 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6823", 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6820", 2024-09-21T23:26:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6821", 2024-09-21T23:26:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6817/1315554966", 2024-09-21T23:26:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6819/1315554966", 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6823/1315554966", 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6821/1315554966", 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:02.661 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:02.661 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:02.661 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.514872+0000", 2024-09-21T23:26:02.661 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:02.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.661 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:02.661 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:02.661 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:02.661 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.912129+0000", 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:02.662 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.745154+0000", 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:16.758871+0000", 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:02.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.287462+0000", 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:02.664 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:18.031636+0000", 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:02.665 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.055855+0000", 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.524609+0000", 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [ 2024-09-21T23:26:02.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: 3, 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: 4 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [], 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [], 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": {}, 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-09-21T23:26:02.667 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-09-21T23:26:02.668 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-09-21T23:26:02.669 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-09-21T23:26:02.669 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-09-21T23:26:02.669 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-09-21T23:26:02.669 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-09-21T23:26:02.669 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.669 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:02.670 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:02.670 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:02.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:02.670 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:02.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.670 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:1 2.3 e19/18 2:c4c92e5a:::rbd_trash:head v 19'1, mlcod=0'0)", 2024-09-21T23:26:02.670 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:02.670 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 34.441504125999998, 2024-09-21T23:26:02.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039243489999999997, 2024-09-21T23:26:02.670 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2999999999999999e-07 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167667+0000", 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7169999999999997e-06 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167679+0000", 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.276e-05 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167688+0000", 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8380000000000008e-06 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167739+0000", 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0964999999999997e-05 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167762+0000", 2024-09-21T23:26:02.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2221999999999998e-05 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171515+0000", 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037531119999999999 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171582+0000", 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7504999999999998e-05 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:02.675 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:02.676 INFO:tasks.thrashosds.thrasher:j is {'epoch': 37, 'fsid': 'd51cc561-ddb8-41ef-b61c-7c6d698e4dd0', 'created': '2024-09-21T23:25:09.056809+0000', 'modified': '2024-09-21T23:26:01.307042+0000', 'last_up_change': '2024-09-21T23:25:20.126996+0000', 'last_in_change': '2024-09-21T23:25:11.226810+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T23:25:19.818212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T23:25:26.968096+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 2339966024}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 2339966024}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 2339966024}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 2339966024}]}, 'public_addr': '172.21.15.31:6817/2339966024', 'cluster_addr': '172.21.15.31:6819/2339966024', 'heartbeat_back_addr': '172.21.15.31:6823/2339966024', 'heartbeat_front_addr': '172.21.15.31:6821/2339966024', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '88ee06a3-9a6a-41e8-9327-a7a61304c03d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 2889649822}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2889649822}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2889649822}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2889649822}]}, 'public_addr': '172.21.15.31:6825/2889649822', 'cluster_addr': '172.21.15.31:6827/2889649822', 'heartbeat_back_addr': '172.21.15.31:6831/2889649822', 'heartbeat_front_addr': '172.21.15.31:6829/2889649822', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 864917651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 864917651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 864917651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 864917651}]}, 'public_addr': '172.21.15.31:6809/864917651', 'cluster_addr': '172.21.15.31:6811/864917651', 'heartbeat_back_addr': '172.21.15.31:6815/864917651', 'heartbeat_front_addr': '172.21.15.31:6813/864917651', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '562e8c30-0622-4cb3-b5d1-f15b42a444bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 36, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6800', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6801', 'nonce': 1713632981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1713632981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1713632981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1713632981}]}, 'public_addr': '172.21.15.31:6801/1713632981', 'cluster_addr': '172.21.15.31:6803/1713632981', 'heartbeat_back_addr': '172.21.15.31:6807/1713632981', 'heartbeat_front_addr': '172.21.15.31:6805/1713632981', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7030ff68-4556-4ad5-abc9-be758f1caa85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6809', 'nonce': 2660753894}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6811', 'nonce': 2660753894}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6815', 'nonce': 2660753894}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6813', 'nonce': 2660753894}]}, 'public_addr': '172.21.15.60:6809/2660753894', 'cluster_addr': '172.21.15.60:6811/2660753894', 'heartbeat_back_addr': '172.21.15.60:6815/2660753894', 'heartbeat_front_addr': '172.21.15.60:6813/2660753894', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7169a163-f067-4385-b734-f58a3ba9e89e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6824', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6825', 'nonce': 2320687649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6826', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6827', 'nonce': 2320687649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6830', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6831', 'nonce': 2320687649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6828', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6829', 'nonce': 2320687649}]}, 'public_addr': '172.21.15.60:6825/2320687649', 'cluster_addr': '172.21.15.60:6827/2320687649', 'heartbeat_back_addr': '172.21.15.60:6831/2320687649', 'heartbeat_front_addr': '172.21.15.60:6829/2320687649', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e511295d-2177-4dcb-a74b-b16b92534df6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6801', 'nonce': 3490482162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6803', 'nonce': 3490482162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6807', 'nonce': 3490482162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6805', 'nonce': 3490482162}]}, 'public_addr': '172.21.15.60:6801/3490482162', 'cluster_addr': '172.21.15.60:6803/3490482162', 'heartbeat_back_addr': '172.21.15.60:6807/3490482162', 'heartbeat_front_addr': '172.21.15.60:6805/3490482162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '544f8a91-ce82-4855-85eb-ea814eb3d058', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6816', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6817', 'nonce': 1315554966}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6818', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6819', 'nonce': 1315554966}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6822', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6823', 'nonce': 1315554966}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6820', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6821', 'nonce': 1315554966}]}, 'public_addr': '172.21.15.60:6817/1315554966', 'cluster_addr': '172.21.15.60:6819/1315554966', 'heartbeat_back_addr': '172.21.15.60:6823/1315554966', 'heartbeat_front_addr': '172.21.15.60:6821/1315554966', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.514872+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.912129+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.745154+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:16.758871+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.287462+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:18.031636+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.055855+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.524609+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [3, 4]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T23:26:02.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:26:02.677 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:02.678 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:02.670+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:02.771 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:02.769+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:02.833 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:02.833 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:02.833 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:02.833 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:02.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:02.884 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:02.883+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:02.972 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:02.972+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:02.998 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:02.998 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:03.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:26:03.029 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:26:03.042 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-21T23:26:01.781598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236060,"kb_used_data":3160,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738660,"statfs":{"total":773094113280,"available":772852387840,"internally_reserved":0,"allocated":3235840,"data_stored":2962849,"data_compressed":27259,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000508"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T23:25:58.965033+0000","last_change":"2024-09-21T23:25:30.180125+0000","last_active":"2024-09-21T23:25:58.965033+0000","last_peered":"2024-09-21T23:25:58.965033+0000","last_clean":"2024-09-21T23:25:58.965033+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:25:58.965033+0000","last_undegraded":"2024-09-21T23:25:58.965033+0000","last_fullsized":"2024-09-21T23:25:58.965033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:17.515296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033759699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":48,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-21T23:26:00.923045+0000","last_change":"2024-09-21T23:25:30.189119+0000","last_active":"2024-09-21T23:26:00.923045+0000","last_peered":"2024-09-21T23:26:00.923045+0000","last_clean":"2024-09-21T23:26:00.923045+0000","last_became_active":"2024-09-21T23:25:28.165264+0000","last_became_peered":"2024-09-21T23:25:28.165264+0000","last_unstale":"2024-09-21T23:26:00.923045+0000","last_undegraded":"2024-09-21T23:26:00.923045+0000","last_fullsized":"2024-09-21T23:26:00.923045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:39:53.806934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053117699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-21T23:26:00.312686+0000","last_change":"2024-09-21T23:25:30.263565+0000","last_active":"2024-09-21T23:26:00.312686+0000","last_peered":"2024-09-21T23:26:00.312686+0000","last_clean":"2024-09-21T23:26:00.312686+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:26:00.312686+0000","last_undegraded":"2024-09-21T23:26:00.312686+0000","last_fullsized":"2024-09-21T23:26:00.312686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:46:37.545348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042085499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-21T23:26:00.922992+0000","last_change":"2024-09-21T23:25:30.189121+0000","last_active":"2024-09-21T23:26:00.922992+0000","last_peered":"2024-09-21T23:26:00.922992+0000","last_clean":"2024-09-21T23:26:00.922992+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:26:00.922992+0000","last_undegraded":"2024-09-21T23:26:00.922992+0000","last_fullsized":"2024-09-21T23:26:00.922992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:00:22.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054857199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":50,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-21T23:26:00.922913+0000","last_change":"2024-09-21T23:25:30.186984+0000","last_active":"2024-09-21T23:26:00.922913+0000","last_peered":"2024-09-21T23:26:00.922913+0000","last_clean":"2024-09-21T23:26:00.922913+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:26:00.922913+0000","last_undegraded":"2024-09-21T23:26:00.922913+0000","last_fullsized":"2024-09-21T23:26:00.922913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:01:38.740017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095847300000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-21T23:26:00.312478+0000","last_change":"2024-09-21T23:25:30.263343+0000","last_active":"2024-09-21T23:26:00.312478+0000","last_peered":"2024-09-21T23:26:00.312478+0000","last_clean":"2024-09-21T23:26:00.312478+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:26:00.312478+0000","last_undegraded":"2024-09-21T23:26:00.312478+0000","last_fullsized":"2024-09-21T23:26:00.312478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:50:16.699815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044148899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-21T23:26:00.312664+0000","last_change":"2024-09-21T23:25:30.263501+0000","last_active":"2024-09-21T23:26:00.312664+0000","last_peered":"2024-09-21T23:26:00.312664+0000","last_clean":"2024-09-21T23:26:00.312664+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:26:00.312664+0000","last_undegraded":"2024-09-21T23:26:00.312664+0000","last_fullsized":"2024-09-21T23:26:00.312664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.155893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036082099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":49,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-21T23:26:00.922921+0000","last_change":"2024-09-21T23:25:30.186477+0000","last_active":"2024-09-21T23:26:00.922921+0000","last_peered":"2024-09-21T23:26:00.922921+0000","last_clean":"2024-09-21T23:26:00.922921+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:26:00.922921+0000","last_undegraded":"2024-09-21T23:26:00.922921+0000","last_fullsized":"2024-09-21T23:26:00.922921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:32:47.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044681799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-21T23:25:55.313788+0000","last_change":"2024-09-21T23:25:21.135362+0000","last_active":"2024-09-21T23:25:55.313788+0000","last_peered":"2024-09-21T23:25:55.313788+0000","last_clean":"2024-09-21T23:25:55.313788+0000","last_became_active":"2024-09-21T23:25:21.135038+0000","last_became_peered":"2024-09-21T23:25:21.135038+0000","last_unstale":"2024-09-21T23:25:55.313788+0000","last_undegraded":"2024-09-21T23:25:55.313788+0000","last_fullsized":"2024-09-21T23:25:55.313788+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:42:58.183351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":27259,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542156,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31632,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":450560,"data_stored":743461,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509453,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":385024,"data_stored":147612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542155,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31620,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":434176,"data_stored":743461,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509452,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":393216,"data_stored":153093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509453,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":397312,"data_stored":153112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509452,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":450560,"data_stored":743461,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31536,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340304,"statfs":{"total":96636764160,"available":96604471296,"internally_reserved":0,"allocated":356352,"data_stored":136560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":368640,"data_stored":142089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:26:03.043 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [1, 7, 3, 4] 2024-09-21T23:26:03.044 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.0', '1', '7', '3', '4'] 2024-09-21T23:26:03.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.0 1 7 3 4 2024-09-21T23:26:03.084 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:03.083+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:03.119 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.817792844, 2024-09-21T23:26:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00081488299999999997, 2024-09-21T23:26:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301851+0000", 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7899999999999998e-07 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302047+0000", 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019639200000000001 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302064+0000", 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7380999999999998e-05 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302080+0000", 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5237e-05 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302153+0000", 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3922000000000001e-05 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302162+0000", 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8170000000000005e-06 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302665+0000", 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050245499999999996 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.8150096260000002, 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00068581899999999995, 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304634+0000", 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5499999999999998e-07 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304796+0000", 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016237700000000001 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304813+0000", 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.732e-05 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304825+0000", 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1302e-05 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304896+0000", 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1729000000000004e-05 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304905+0000", 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8359999999999998e-06 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.305319+0000", 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00041360000000000002 2024-09-21T23:26:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:03.128 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:03.129 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:03.174 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:03.174+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:03.249 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:03.249 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:03.249 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:03.249 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:03.258 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:03.299 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:03.299+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:03.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T23:26:03.376 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:03.376+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:03.406 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:03.407 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:03.407 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:03.407 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:03.407 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:03.415 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:03.477 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:03.477+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.261109577, 2024-09-21T23:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00108676, 2024-09-21T23:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301959+0000", 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3099999999999996e-07 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302322+0000", 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00036268299999999998 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302345+0000", 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3278e-05 2024-09-21T23:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302367+0000", 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2118999999999999e-05 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302468+0000", 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100579 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302479+0000", 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1430999999999999e-05 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.303045+0000", 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000565739 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.2583533390000001, 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010194869999999999, 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304715+0000", 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9599999999999998e-07 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305026+0000", 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031072100000000002 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305048+0000", 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1875999999999999e-05 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305066+0000", 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8644e-05 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305163+0000", 2024-09-21T23:26:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6333000000000001e-05 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305172+0000", 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2189999999999993e-06 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305734+0000", 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056179799999999996 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:03.572 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:03.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:03.616 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:03.616+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:03.678 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:03.678+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:03.680 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:03.680 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:03.680 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:03.680 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:03.689 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:03.779 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:03.779+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:03.837 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:03.837 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:03.838 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:03.838 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:03.838 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:03.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:03.887 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:03.887+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:03.980 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:03.980+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:03.994 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:03.994 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:03.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:03.994 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 35.830378615000001, 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0080986310000000002, 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.995 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:03.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163709+0000", 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163727+0000", 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7552000000000001e-05 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163744+0000", 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7541999999999999e-05 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:03.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163757+0000", 2024-09-21T23:26:03.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2855e-05 2024-09-21T23:26:03.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:03.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164069+0000", 2024-09-21T23:26:03.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031179899999999997 2024-09-21T23:26:03.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164084+0000", 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5265e-05 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167127+0000", 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030427029999999999 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167317+0000", 2024-09-21T23:26:03.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000190006 2024-09-21T23:26:04.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:26:04.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167465+0000", 2024-09-21T23:26:04.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014824 2024-09-21T23:26:04.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:04.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.168309+0000", 2024-09-21T23:26:04.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084350899999999999 2024-09-21T23:26:04.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171730+0000", 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034217620000000001 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171770+0000", 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0074999999999998e-05 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:04.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171802+0000", 2024-09-21T23:26:04.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1563999999999998e-05 2024-09-21T23:26:04.002 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.002 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:04.002 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.002 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:1 2.3 e19/18 ondisk, result = 0)", 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 35.822436367999998, 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016281800000000001, 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6899999999999996e-07 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171651+0000", 2024-09-21T23:26:04.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0309999999999998e-06 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171665+0000", 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.309e-05 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171672+0000", 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6070000000000002e-06 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171716+0000", 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4344000000000001e-05 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171727+0000", 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0488e-05 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171808+0000", 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1489000000000005e-05 2024-09-21T23:26:04.006 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 35.819660925999997, 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023803299999999999, 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:04.007 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 2 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174422+0000", 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4300000000000001e-06 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174429+0000", 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1659999999999999e-06 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174443+0000", 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4358e-05 2024-09-21T23:26:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174449+0000", 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2770000000000002e-06 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174540+0000", 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0209000000000003e-05 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174659+0000", 2024-09-21T23:26:04.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011959300000000001 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 35.819195925999999, 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016795100000000001, 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:04.011 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:04.012 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:04.012 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:04.012 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 3 2024-09-21T23:26:04.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.012 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:04.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:04.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:04.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:04.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:04.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:04.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174887+0000", 2024-09-21T23:26:04.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-09-21T23:26:04.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:04.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174892+0000", 2024-09-21T23:26:04.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8180000000000003e-06 2024-09-21T23:26:04.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174902+0000", 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0095000000000001e-05 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174911+0000", 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0799999999999995e-06 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174974+0000", 2024-09-21T23:26:04.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3027000000000005e-05 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.175054+0000", 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0011000000000004e-05 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.825690295000001, 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006147263, 2024-09-21T23:26:04.016 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:04.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168394+0000", 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.956e-06 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:04.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168404+0000", 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0463e-05 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168423+0000", 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8715999999999998e-05 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168432+0000", 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.516e-06 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168529+0000", 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7073999999999999e-05 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168534+0000", 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.831e-06 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170224+0000", 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016893590000000001 2024-09-21T23:26:04.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170332+0000", 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108346 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170445+0000", 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112642 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.171210+0000", 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076543800000000001 2024-09-21T23:26:04.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174469+0000", 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003259086 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174506+0000", 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6465999999999999e-05 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:04.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174539+0000", 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3370000000000001e-05 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:5 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.819748250000004, 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021298500000000001, 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:04.023 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174335+0000", 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.111e-06 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174341+0000", 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6239999999999996e-06 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174356+0000", 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4548e-05 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174375+0000", 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8757e-05 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174457+0000", 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2702999999999997e-05 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174466+0000", 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.208e-06 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174547+0000", 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1033999999999997e-05 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 33.809976210999999, 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040438610000000002, 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:04.027 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184109+0000", 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9069999999999999e-06 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184124+0000", 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5780000000000001e-05 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184159+0000", 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4106999999999997e-05 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184178+0000", 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9908999999999999e-05 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184330+0000", 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015152300000000001 2024-09-21T23:26:04.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184337+0000", 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7920000000000004e-06 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186239+0000", 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019020129999999999 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186307+0000", 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8326999999999998e-05 2024-09-21T23:26:04.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186459+0000", 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152099 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187261+0000", 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00080186199999999997 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188085+0000", 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00082417199999999997 2024-09-21T23:26:04.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188120+0000", 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5194999999999998e-05 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188150+0000", 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9175000000000001e-05 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.033 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:7 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 33.806105475000003, 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000179505, 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.16e-07 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187985+0000", 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3039999999999999e-06 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188000+0000", 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5097e-05 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188007+0000", 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1330000000000001e-06 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188074+0000", 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7514000000000004e-05 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188082+0000", 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3649999999999998e-06 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188156+0000", 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4176000000000003e-05 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:04.037 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:04.038 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:04.081 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:04.080+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:04.146 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:04.146 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:04.146 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:04.146 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:04.155 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:04.181 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:04.181+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:04.261 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:04.262 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:04.262 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:04.262 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:04.262 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:04.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:04.282 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:04.282+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:set 2.0 pg_upmap_items mapping to [1->7,3->4] 2024-09-21T23:26:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-09-21T23:26:04.359 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:26:04.378 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:04.378 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:04.378 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:04.378 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-09-21T23:26:04.378 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:04.382 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:04.382+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:04.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:04.484 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:04.484+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:04.519 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:04.535 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:04.535 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:04.535 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:04.535 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:04.544 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:04.550 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:04.551 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:04.567 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:04.568 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:04.584 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:04.584 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:04.589 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:04.589+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:04.600 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:04.601 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:04.617 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:04.617 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:04.634 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:04.634 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:04.651 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:04.651 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:04.651 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:04.652 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:04.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:04.703 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:04.703+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:04.785 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:04.785+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:04.807 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:04.808 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:04.817 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:04.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:04.886+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:04.926 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:04.926 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:04.935 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:04.987 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:04.987+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:05.087 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:05.087+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:05.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:05.188 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:05.188+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:05.255 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:05.255 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:05.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:05.288 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:05.288+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:05.378 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:05.378 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:05.389 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:05.389 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:05.388+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:05.489 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:05.489+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:05.504 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:05.504 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:05.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:05.590 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:05.590+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:05.629 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:05.630 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:05.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:05.691 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:05.691+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:05.763 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:05.763 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:05.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:05.816 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:05.816+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:05.881 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:05.881 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:05.890 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:05.892 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:05.892+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:05.993 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:05.993+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:06.041 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:06.042 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:06.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:06.094 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:06.094+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:06.161 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:06.161 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:06.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:06.195 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:06.195+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:06.278 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:06.278 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:06.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:06.330 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:06.330+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:06.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T23:26:06.397 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:06.397+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:06.400 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:06.400 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:06.409 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:06.498 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:06.498+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:06.530 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:06.530 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:06.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:06.624 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:06.624+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:06.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:26:06.689 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:06.689 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:06.698 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:06.706 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:06.706+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:06.802 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:06.802+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:06.810 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:06.810 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:06.829 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:06.830 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:06.844 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:06.845 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:06.845 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:06.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:06.860 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:06.861 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:06.877 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:06.877 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:06.908 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:06.908 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:06.939 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:06.939 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:06.940 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:06.940+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:06.956 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:07.003 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:07.003 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:07.004 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:07.004 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:07.004 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:07.004+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:07.012 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:07.105 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:07.104+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:07.159 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:07.159 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:07.159 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:07.159 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:07.159 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:07.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:07.205 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:07.205+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:07.275 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:07.275 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:07.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:07.275 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 39.111521498000002, 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0080986310000000002, 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:07.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163709+0000", 2024-09-21T23:26:07.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163727+0000", 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7552000000000001e-05 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163744+0000", 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7541999999999999e-05 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163757+0000", 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2855e-05 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164069+0000", 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031179899999999997 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164084+0000", 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5265e-05 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167127+0000", 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030427029999999999 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167317+0000", 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000190006 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167465+0000", 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014824 2024-09-21T23:26:07.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.168309+0000", 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084350899999999999 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171730+0000", 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034217620000000001 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171770+0000", 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0074999999999998e-05 2024-09-21T23:26:07.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171802+0000", 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1563999999999998e-05 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:1 2.3 e19/18 ondisk, result = 0)", 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 39.103579250999999, 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016281800000000001, 2024-09-21T23:26:07.282 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6899999999999996e-07 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171651+0000", 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0309999999999998e-06 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171665+0000", 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.309e-05 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171672+0000", 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6070000000000002e-06 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171716+0000", 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4344000000000001e-05 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171727+0000", 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0488e-05 2024-09-21T23:26:07.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171808+0000", 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1489000000000005e-05 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 39.100803808999999, 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023803299999999999, 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:07.286 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 2 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174422+0000", 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4300000000000001e-06 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174429+0000", 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1659999999999999e-06 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174443+0000", 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4358e-05 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174449+0000", 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2770000000000002e-06 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174540+0000", 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0209000000000003e-05 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174659+0000", 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011959300000000001 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 39.100338809, 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016795100000000001, 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:07.290 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 3 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:07.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174887+0000", 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174892+0000", 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8180000000000003e-06 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174902+0000", 2024-09-21T23:26:07.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0095000000000001e-05 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174911+0000", 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0799999999999995e-06 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174974+0000", 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3027000000000005e-05 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:07.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.175054+0000", 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0011000000000004e-05 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 38.106833178000002, 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006147263, 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:07.294 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:07.295 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:07.295 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:07.295 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5 2024-09-21T23:26:07.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.295 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:07.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:07.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:07.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168394+0000", 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.956e-06 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168404+0000", 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0463e-05 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168423+0000", 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8715999999999998e-05 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168432+0000", 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.516e-06 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168529+0000", 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7073999999999999e-05 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168534+0000", 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.831e-06 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170224+0000", 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016893590000000001 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170332+0000", 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108346 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170445+0000", 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112642 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.171210+0000", 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076543800000000001 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174469+0000", 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003259086 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174506+0000", 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6465999999999999e-05 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174539+0000", 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3370000000000001e-05 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:5 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:07.300 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 38.100891132999998, 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021298500000000001, 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:07.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174335+0000", 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.111e-06 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174341+0000", 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6239999999999996e-06 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:07.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174356+0000", 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4548e-05 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174375+0000", 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8757e-05 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174457+0000", 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2702999999999997e-05 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174466+0000", 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.208e-06 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174547+0000", 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1033999999999997e-05 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-21T23:26:07.304 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 37.091119094, 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040438610000000002, 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:07.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184109+0000", 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9069999999999999e-06 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:07.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184124+0000", 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5780000000000001e-05 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184159+0000", 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4106999999999997e-05 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184178+0000", 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9908999999999999e-05 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184330+0000", 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015152300000000001 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184337+0000", 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7920000000000004e-06 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186239+0000", 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019020129999999999 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186307+0000", 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8326999999999998e-05 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186459+0000", 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152099 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187261+0000", 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00080186199999999997 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:07.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188085+0000", 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00082417199999999997 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188120+0000", 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5194999999999998e-05 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188150+0000", 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9175000000000001e-05 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:7 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 37.087248357999997, 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000179505, 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.16e-07 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187985+0000", 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3039999999999999e-06 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188000+0000", 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5097e-05 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188007+0000", 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1330000000000001e-06 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188074+0000", 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7514000000000004e-05 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188082+0000", 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3649999999999998e-06 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188156+0000", 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4176000000000003e-05 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:07.314 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:07.316 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:07.316 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:07.316+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:07.339 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-09-21T23:26:07.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:07.416+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:07.464 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:07.464 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:07.464 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:07.464 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:07.473 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:07.517 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:07.517+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:07.581 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:07.581 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:07.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:07.630 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:07.630+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:07.720 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:07.719+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:07.736 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:07.736 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:07.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:07.737 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-09-21T23:26:07.737 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:07.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:07.821 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:07.820+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:07.853 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:07.853 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:07.853 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:07.853 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:07.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:07.922 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:07.921+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:07.969 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:07.970 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:07.970 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:07.970 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:07.970 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:07.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:08.022 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:08.022+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:08.087 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.918986789999998, 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090066050000000009, 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.088 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 37 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7099999999999998e-07 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167009+0000", 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.92e-06 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167013+0000", 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 2024-09-21T23:26:08.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167017+0000", 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7280000000000001e-06 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170778+0000", 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037612399999999999 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170831+0000", 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2694000000000001e-05 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170970+0000", 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000138966 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.171555+0000", 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058538399999999999 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175872+0000", 2024-09-21T23:26:08.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043172389999999996 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175969+0000", 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6844999999999999e-05 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.176012+0000", 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3356999999999998e-05 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.093 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.892881954000003, 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069023069999999999, 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0999999999999999e-07 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193115+0000", 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5700000000000001e-06 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193129+0000", 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4124000000000001e-05 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193133+0000", 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0550000000000001e-06 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194144+0000", 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001010735 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194231+0000", 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7665999999999997e-05 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194357+0000", 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125353 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194844+0000", 2024-09-21T23:26:08.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000487305 2024-09-21T23:26:08.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199964+0000", 2024-09-21T23:26:08.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051204859999999996 2024-09-21T23:26:08.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199992+0000", 2024-09-21T23:26:08.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7902000000000002e-05 2024-09-21T23:26:08.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.200013+0000", 2024-09-21T23:26:08.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0601000000000001e-05 2024-09-21T23:26:08.100 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.100 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.100 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.100 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.100 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:08.100 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.871232880000001, 2024-09-21T23:26:08.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0067494620000000003, 2024-09-21T23:26:08.100 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 73 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4900000000000001e-07 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214818+0000", 2024-09-21T23:26:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7754000000000003e-05 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214842+0000", 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4552000000000001e-05 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214846+0000", 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9129999999999996e-06 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215813+0000", 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00096695899999999998 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215904+0000", 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0328000000000007e-05 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216020+0000", 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115838 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217018+0000", 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099836200000000003 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221461+0000", 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044435969999999997 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221487+0000", 2024-09-21T23:26:08.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5806e-05 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221509+0000", 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1903999999999999e-05 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.871138176999999, 2024-09-21T23:26:08.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068139230000000004, 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 74 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214855+0000", 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-07 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214859+0000", 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1389999999999997e-06 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214871+0000", 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1698999999999999e-05 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214887+0000", 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6061000000000001e-05 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216359+0000", 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014720779999999999 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216439+0000", 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0191999999999994e-05 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216579+0000", 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014029099999999999 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217129+0000", 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054932099999999997 2024-09-21T23:26:08.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221559+0000", 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044299710000000004 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221627+0000", 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8631000000000001e-05 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221668+0000", 2024-09-21T23:26:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0988000000000001e-05 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.854461991000001, 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0077112509999999997, 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.112 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 86 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5199999999999998e-07 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231533+0000", 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3879999999999998e-06 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231537+0000", 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0010000000000003e-06 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231540+0000", 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0489999999999999e-06 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233890+0000", 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023500249999999999 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233937+0000", 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6400999999999998e-05 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234015+0000", 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8343000000000005e-05 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235675+0000", 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001660124 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239138+0000", 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034624109999999999 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239200+0000", 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2698000000000002e-05 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239242+0000", 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1458999999999997e-05 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.854445722000001, 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010200097999999999, 2024-09-21T23:26:08.117 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2700000000000001e-07 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231588+0000", 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0985000000000001e-05 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231593+0000", 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1059999999999999e-06 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231606+0000", 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2343e-05 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234406+0000", 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027999850000000001 2024-09-21T23:26:08.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234489+0000", 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3051000000000002e-05 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234631+0000", 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285799999999999 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235679+0000", 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001047517 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241644+0000", 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059652129999999996 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241704+0000", 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9732999999999999e-05 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241747+0000", 2024-09-21T23:26:08.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-05 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.854379883, 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010246534, 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.123 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 88 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0199999999999998e-07 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231615+0000", 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3439999999999999e-06 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231619+0000", 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0219999999999998e-06 2024-09-21T23:26:08.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231622+0000", 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6989999999999999e-06 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235337+0000", 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037150030000000001 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235385+0000", 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7861999999999997e-05 2024-09-21T23:26:08.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235458+0000", 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2596999999999996e-05 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235978+0000", 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052013600000000004 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241799+0000", 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058215289999999998 2024-09-21T23:26:08.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241836+0000", 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6282999999999998e-05 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241859+0000", 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3757e-05 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.128 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.696988699999999, 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072206969999999999, 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 112 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.72e-07 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389069+0000", 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4875000000000005e-05 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389079+0000", 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0135e-05 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389085+0000", 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394006+0000", 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049207920000000002 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394052+0000", 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5810999999999999e-05 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394139+0000", 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7130000000000001e-05 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395407+0000", 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012681260000000001 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396139+0000", 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000731703 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396194+0000", 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5309999999999997e-05 2024-09-21T23:26:08.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396225+0000", 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0589000000000001e-05 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.696881742999999, 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086923490000000003, 2024-09-21T23:26:08.134 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 113 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389116+0000", 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.245e-06 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389126+0000", 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0295000000000001e-05 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389132+0000", 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0889999999999996e-06 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394548+0000", 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005415876 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394642+0000", 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4531999999999995e-05 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394714+0000", 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1753000000000002e-05 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396233+0000", 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015185299999999999 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397749+0000", 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015157770000000001 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397779+0000", 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0561000000000001e-05 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397803+0000", 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4116000000000001e-05 2024-09-21T23:26:08.139 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.696849673999999, 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.01139824, 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.140 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 114 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:08.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.01e-07 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389146+0000", 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5239999999999999e-06 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389150+0000", 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3989999999999997e-06 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389153+0000", 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.038e-06 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395026+0000", 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058731720000000003 2024-09-21T23:26:08.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395071+0000", 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4388999999999998e-05 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395141+0000", 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0053000000000002e-05 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397196+0000", 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020555899999999999 2024-09-21T23:26:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400494+0000", 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032972679999999999 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400520+0000", 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6423000000000001e-05 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400541+0000", 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0982999999999999e-05 2024-09-21T23:26:08.145 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.696830994999999, 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011537824, 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.146 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 115 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4900000000000001e-07 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389197+0000", 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5204999999999999e-05 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389203+0000", 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0979999999999999e-06 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389207+0000", 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6600000000000001e-06 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395443+0000", 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062358099999999996 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395525+0000", 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1855999999999997e-05 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395669+0000", 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014472399999999999 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397204+0000", 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001534817 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400599+0000", 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033944499999999998 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400655+0000", 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6647000000000002e-05 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400699+0000", 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4208000000000001e-05 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.696779542999998, 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011571580999999999, 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 116 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.152 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3099999999999999e-07 2024-09-21T23:26:08.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389216+0000", 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4260000000000002e-06 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389223+0000", 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389225+0000", 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7760000000000002e-06 2024-09-21T23:26:08.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396256+0000", 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070309020000000003 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396348+0000", 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1780000000000006e-05 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396481+0000", 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000133101 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397208+0000", 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072716199999999995 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400737+0000", 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035290220000000001 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400763+0000", 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5219999999999999e-05 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400785+0000", 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2045e-05 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.157 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.69676286, 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011687858000000001, 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 117 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3699999999999999e-07 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389232+0000", 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1789999999999998e-06 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389238+0000", 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209999999999999e-06 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389241+0000", 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.557e-06 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396792+0000", 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075506899999999997 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396838+0000", 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5639000000000001e-05 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396958+0000", 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119892 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397458+0000", 2024-09-21T23:26:08.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049987600000000001 2024-09-21T23:26:08.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.162 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400824+0000", 2024-09-21T23:26:08.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033662560000000002 2024-09-21T23:26:08.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.162 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400880+0000", 2024-09-21T23:26:08.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5779999999999998e-05 2024-09-21T23:26:08.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400918+0000", 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8031000000000002e-05 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.684744385999998, 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006886924, 2024-09-21T23:26:08.163 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 120 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401249+0000", 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.13e-07 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401286+0000", 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6773999999999997e-05 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401295+0000", 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9039999999999998e-06 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401299+0000", 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3250000000000001e-06 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402183+0000", 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000883758 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402264+0000", 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1080000000000003e-05 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402401+0000", 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013719700000000001 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403430+0000", 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010290119999999999 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408045+0000", 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046146659999999999 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408095+0000", 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0898e-05 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408135+0000", 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9697000000000003e-05 2024-09-21T23:26:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.684685911999999, 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069210920000000002, 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 121 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3200000000000001e-07 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401341+0000", 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4236999999999998e-05 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401346+0000", 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.04e-06 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401350+0000", 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6399999999999999e-06 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402733+0000", 2024-09-21T23:26:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013825790000000001 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402816+0000", 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3461000000000006e-05 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402941+0000", 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012518300000000001 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403533+0000", 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000592071 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408174+0000", 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046404209999999996 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408204+0000", 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0147000000000001e-05 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408228+0000", 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3981e-05 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.248543015000003, 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064924400000000004, 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 139 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0800000000000001e-07 2024-09-21T23:26:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837452+0000", 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7209999999999999e-06 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837455+0000", 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2279999999999998e-06 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837458+0000", 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8200000000000001e-06 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842187+0000", 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047298460000000002 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842233+0000", 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6056000000000001e-05 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842304+0000", 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0862e-05 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843291+0000", 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00098632100000000007 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843894+0000", 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060369199999999997 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843922+0000", 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7429000000000001e-05 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843942+0000", 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0256999999999999e-05 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.248531071999999, 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066155550000000004, 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 140 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1e-07 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837517+0000", 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5034000000000002e-05 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837522+0000", 2024-09-21T23:26:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8520000000000003e-06 2024-09-21T23:26:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837525+0000", 2024-09-21T23:26:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4139999999999998e-06 2024-09-21T23:26:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842562+0000", 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050366279999999996 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842615+0000", 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3675999999999999e-05 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842715+0000", 2024-09-21T23:26:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9990999999999999e-05 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843873+0000", 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011573 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844026+0000", 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015368900000000001 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844058+0000", 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1449999999999999e-05 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844077+0000", 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9310999999999999e-05 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.248458878999998, 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068324850000000001, 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 141 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.22e-07 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837539+0000", 2024-09-21T23:26:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.023e-06 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837543+0000", 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6490000000000001e-06 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837546+0000", 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.382e-06 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843479+0000", 2024-09-21T23:26:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059328189999999998 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843519+0000", 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0071000000000003e-05 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843600+0000", 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1376000000000002e-05 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844137+0000", 2024-09-21T23:26:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053640299999999997 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844319+0000", 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000181893 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844346+0000", 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.781e-05 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844366+0000", 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9737000000000001e-05 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.633125184000001, 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006594145, 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 158 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452868+0000", 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9500000000000003e-07 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452938+0000", 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9598999999999996e-05 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452950+0000", 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2971e-05 2024-09-21T23:26:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452960+0000", 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.448e-06 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456217+0000", 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003257378 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456557+0000", 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033989399999999998 2024-09-21T23:26:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456703+0000", 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000145599 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459339+0000", 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026365529999999998 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459386+0000", 2024-09-21T23:26:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6640000000000001e-05 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459428+0000", 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2148999999999999e-05 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459462+0000", 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3419e-05 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.633020809999998, 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0065672760000000004, 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 159 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5400000000000001e-07 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452977+0000", 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.301e-06 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452984+0000", 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1840000000000002e-06 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452989+0000", 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5219999999999998e-06 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457324+0000", 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043348780000000003 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457416+0000", 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1824999999999996e-05 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457574+0000", 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015810999999999999 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459376+0000", 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001801649 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459467+0000", 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1367999999999997e-05 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459507+0000", 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9576000000000002e-05 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459539+0000", 2024-09-21T23:26:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2308999999999998e-05 2024-09-21T23:26:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:08.205 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:08.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:08.207 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:08.206+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:08.307 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:08.307+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:08.330 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:08.330 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:08.330 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:08.330 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:08.339 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:08.424 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:08.424+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:08.501 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:08.501 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:08.501 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:08.501 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:08.501 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:08.509 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:08.509+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:08.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:08.611 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:08.611+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:08.663 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:08.663 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:08.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:08.663 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:08.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.663 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-09-21T23:26:08.663 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:08.663 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.524438224000001, 2024-09-21T23:26:08.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036127939999999999, 2024-09-21T23:26:08.663 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.663 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3500000000000002e-07 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136608+0000", 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.182e-06 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136618+0000", 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9199999999999999e-06 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136628+0000", 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4390000000000006e-06 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136692+0000", 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3979000000000005e-05 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136716+0000", 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4391000000000001e-05 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140171+0000", 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034554910000000002 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140214+0000", 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2057000000000001e-05 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'18)", 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:08.667 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.489920060999999, 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043957730000000004, 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171120+0000", 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6700000000000003e-07 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171125+0000", 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.451e-06 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171138+0000", 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2541000000000001e-05 2024-09-21T23:26:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171143+0000", 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4659999999999998e-06 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171187+0000", 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4014000000000002e-05 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171200+0000", 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3108999999999999e-05 2024-09-21T23:26:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175479+0000", 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042784570000000003 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175515+0000", 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5967999999999999e-05 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:58 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-21T23:26:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.467167764999999, 2024-09-21T23:26:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003032851, 2024-09-21T23:26:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193872+0000", 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9400000000000004e-07 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193952+0000", 2024-09-21T23:26:08.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9791999999999998e-05 2024-09-21T23:26:08.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193962+0000", 2024-09-21T23:26:08.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0034000000000001e-05 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193973+0000", 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0733e-05 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194015+0000", 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2911999999999998e-05 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194031+0000", 2024-09-21T23:26:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5082999999999999e-05 2024-09-21T23:26:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196869+0000", 2024-09-21T23:26:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002838808 2024-09-21T23:26:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196904+0000", 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4694999999999999e-05 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:59 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.466582099, 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053675709999999998, 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7100000000000002e-07 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194462+0000", 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.668e-06 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194469+0000", 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1670000000000002e-06 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194475+0000", 2024-09-21T23:26:08.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-06 2024-09-21T23:26:08.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194513+0000", 2024-09-21T23:26:08.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7391999999999997e-05 2024-09-21T23:26:08.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194531+0000", 2024-09-21T23:26:08.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7870999999999999e-05 2024-09-21T23:26:08.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.682 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.682 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199791+0000", 2024-09-21T23:26:08.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052604119999999999 2024-09-21T23:26:08.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199824+0000", 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3340000000000003e-05 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:60 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'38, mlcod=16'36)", 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.460501403999999, 2024-09-21T23:26:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003614562, 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200539+0000", 2024-09-21T23:26:08.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5799999999999999e-06 2024-09-21T23:26:08.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200551+0000", 2024-09-21T23:26:08.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1816e-05 2024-09-21T23:26:08.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200578+0000", 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6772999999999999e-05 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200590+0000", 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2303000000000001e-05 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200715+0000", 2024-09-21T23:26:08.686 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012523499999999999 2024-09-21T23:26:08.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200731+0000", 2024-09-21T23:26:08.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5690000000000001e-05 2024-09-21T23:26:08.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.687 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.687 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204121+0000", 2024-09-21T23:26:08.687 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003390363 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204152+0000", 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0802000000000001e-05 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:70 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'39)", 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:08.688 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.450445748999996, 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033896579999999998, 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210594+0000", 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1099999999999995e-07 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210600+0000", 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4389999999999997e-06 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210613+0000", 2024-09-21T23:26:08.690 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.243e-05 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210618+0000", 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0939999999999998e-06 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210843+0000", 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022493200000000001 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210858+0000", 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5109000000000001e-05 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213938+0000", 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030796500000000002 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213983+0000", 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5293000000000002e-05 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:72 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'46, mlcod=16'43)", 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.445500973000001, 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029102279999999999, 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215539+0000", 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4199999999999995e-07 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215589+0000", 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0488999999999997e-05 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215598+0000", 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2830000000000007e-06 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215610+0000", 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1134999999999999e-05 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215657+0000", 2024-09-21T23:26:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7658999999999999e-05 2024-09-21T23:26:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215679+0000", 2024-09-21T23:26:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1526999999999999e-05 2024-09-21T23:26:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218410+0000", 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027316939999999998 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218448+0000", 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7799000000000002e-05 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:73 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'47, mlcod=16'43)", 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.444891626, 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051882020000000003, 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.698 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:08.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.699 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.699 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216148+0000", 2024-09-21T23:26:08.699 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7100000000000001e-07 2024-09-21T23:26:08.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216218+0000", 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0313000000000003e-05 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216227+0000", 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9959999999999999e-06 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.700 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216232+0000", 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5340000000000001e-06 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216267+0000", 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4739000000000001e-05 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216281+0000", 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4345000000000001e-05 2024-09-21T23:26:08.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221290+0000", 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050088989999999998 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221336+0000", 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5704999999999997e-05 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:74 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'48, mlcod=16'43)", 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.444365683000001, 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046922939999999996, 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216674+0000", 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9900000000000001e-07 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216679+0000", 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.502e-06 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216687+0000", 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1790000000000008e-06 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216691+0000", 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7230000000000002e-06 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216737+0000", 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6087000000000001e-05 2024-09-21T23:26:08.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216758+0000", 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0434000000000001e-05 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221338+0000", 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045804569999999996 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221366+0000", 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7413e-05 2024-09-21T23:26:08.706 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:86 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'56, mlcod=16'50)", 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.426943749000003, 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049055269999999998, 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234096+0000", 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234099+0000", 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3890000000000001e-06 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234107+0000", 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2349999999999999e-06 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234112+0000", 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0629999999999999e-06 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234143+0000", 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1131999999999999e-05 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234155+0000", 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2140000000000001e-05 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.238972+0000", 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048167770000000004 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.239001+0000", 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9164000000000001e-05 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:87 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'57, mlcod=16'52)", 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.426264373999999, 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067179910000000004, 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.711 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.712 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0299999999999998e-07 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234813+0000", 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7531999999999999e-05 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234826+0000", 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2617e-05 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.713 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234831+0000", 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1229999999999999e-06 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234869+0000", 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8513999999999998e-05 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234881+0000", 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2148e-05 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.714 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241465+0000", 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065837329999999996 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241493+0000", 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7521e-05 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:88 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:08.715 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.425466526000001, 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059344530000000001, 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235573+0000", 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0100000000000004e-07 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235578+0000", 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1590000000000001e-06 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235590+0000", 2024-09-21T23:26:08.717 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1229e-05 2024-09-21T23:26:08.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235595+0000", 2024-09-21T23:26:08.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0889999999999999e-06 2024-09-21T23:26:08.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.718 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.718 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235943+0000", 2024-09-21T23:26:08.718 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000348227 2024-09-21T23:26:08.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235957+0000", 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4419999999999999e-05 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241494+0000", 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055371070000000003 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241507+0000", 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2520999999999999e-05 2024-09-21T23:26:08.719 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:113 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'75, mlcod=16'71)", 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.266235248000001, 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028543489999999999, 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.720 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8199999999999999e-07 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394809+0000", 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394821+0000", 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2024999999999999e-05 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394827+0000", 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0020000000000003e-06 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394862+0000", 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4811000000000003e-05 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394873+0000", 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.058e-05 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397631+0000", 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027581060000000002 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397658+0000", 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7073999999999999e-05 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'71)", 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.265823834999999, 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005160085, 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.724 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395216+0000", 2024-09-21T23:26:08.725 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4999999999999998e-07 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395221+0000", 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2290000000000002e-06 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395229+0000", 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2369999999999992e-06 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.726 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395233+0000", 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1840000000000001e-06 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395260+0000", 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.728e-05 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395272+0000", 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1270000000000001e-05 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.727 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400341+0000", 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050691809999999999 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400375+0000", 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4254000000000001e-05 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'72)", 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:08.728 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.265275314, 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046274580000000001, 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395829+0000", 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4618999999999999e-05 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395836+0000", 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9090000000000003e-06 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395839+0000", 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.371e-06 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395868+0000", 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9328000000000001e-05 2024-09-21T23:26:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395880+0000", 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1501000000000001e-05 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400376+0000", 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044966340000000002 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400391+0000", 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4676e-05 2024-09-21T23:26:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:116 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'78, mlcod=16'73)", 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.264470766999999, 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038400970000000002, 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396569+0000", 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1100000000000001e-07 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396573+0000", 2024-09-21T23:26:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5569999999999997e-06 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396579+0000", 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1120000000000001e-06 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396587+0000", 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396630+0000", 2024-09-21T23:26:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2991000000000002e-05 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396642+0000", 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1481000000000001e-05 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400392+0000", 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037507830000000002 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.736 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400408+0000", 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5863e-05 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:117 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'79, mlcod=16'73)", 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.263968425000002, 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033512120000000001, 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:08.738 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7e-07 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397075+0000", 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7809999999999999e-06 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397082+0000", 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.739 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397085+0000", 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7139999999999999e-06 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397130+0000", 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.498e-05 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397142+0000", 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1838000000000001e-05 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.740 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400409+0000", 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032666349999999999 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400422+0000", 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2866e-05 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.259144751999997, 2024-09-21T23:26:08.741 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034761240000000001, 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401895+0000", 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7400000000000003e-07 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401963+0000", 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8701000000000005e-05 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401972+0000", 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9530000000000005e-06 2024-09-21T23:26:08.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401977+0000", 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1579999999999998e-06 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402009+0000", 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2512000000000003e-05 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402022+0000", 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3220999999999999e-05 2024-09-21T23:26:08.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405337+0000", 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033150300000000001 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405370+0000", 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2975000000000002e-05 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.745 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.258538682000001, 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054168369999999999, 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.746 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402501+0000", 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2699999999999999e-07 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402589+0000", 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899000000000007e-05 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.747 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402599+0000", 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0312000000000001e-05 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402605+0000", 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9689999999999999e-06 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402648+0000", 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3210000000000001e-05 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402664+0000", 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5347999999999999e-05 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407887+0000", 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052237919999999997 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407917+0000", 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9779999999999999e-05 2024-09-21T23:26:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.258009301000001, 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049049180000000003, 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.750 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4099999999999999e-07 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403098+0000", 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7464000000000003e-05 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403106+0000", 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6629999999999999e-06 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403111+0000", 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403137+0000", 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6349000000000001e-05 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403148+0000", 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.077e-05 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407920+0000", 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047716590000000001 2024-09-21T23:26:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:08.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:08.754 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:08.754 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407935+0000", 2024-09-21T23:26:08.754 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4783999999999999e-05 2024-09-21T23:26:08.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.754 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:08.754 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:08.754 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:08.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:08.756 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:08.756+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:08.857 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:08.856+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:08.910 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:08.910 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:08.910 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:08.911 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:08.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:08.963 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:08.963+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:08.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:26:09.059 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:09.059+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:09.117 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:09.117 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:09.117 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:09.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:09.160 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:09.160+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:09.169 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:09.169 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:09.203 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:09.203 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:09.235 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:09.235 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:09.252 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:09.252 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:09.256 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:09.256 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:09.260 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:09.260+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:09.265 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:09.280 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:09.281 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:09.306 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:09.306 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:09.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:09.326 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:09.348 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:26:09.349 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:26:09.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:26:09.352 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:09.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T23:26:09.369 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:09.369+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:09.438 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:09.438 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:09.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:09.463 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:09.463+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:09.563 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:09.563+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:09.609 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:09.610 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:09.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:09.664 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:09.664+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:09.765 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:09.765+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:26:09.773 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:26:09.788 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-09-21T23:26:07.782593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236188,"kb_used_data":3316,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738532,"statfs":{"total":773094113280,"available":772852256768,"internally_reserved":0,"allocated":3395584,"data_stored":2476995,"data_compressed":19550,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.473591"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-21T23:26:02.465806+0000","last_change":"2024-09-21T23:25:30.180125+0000","last_active":"2024-09-21T23:26:02.465806+0000","last_peered":"2024-09-21T23:26:02.465806+0000","last_clean":"2024-09-21T23:26:02.465806+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:26:02.465806+0000","last_undegraded":"2024-09-21T23:26:02.465806+0000","last_fullsized":"2024-09-21T23:26:02.465806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:17.515296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033759699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-21T23:26:04.331866+0000","last_change":"2024-09-21T23:25:30.189119+0000","last_active":"2024-09-21T23:26:04.331866+0000","last_peered":"2024-09-21T23:26:04.331866+0000","last_clean":"2024-09-21T23:26:04.331866+0000","last_became_active":"2024-09-21T23:25:28.165264+0000","last_became_peered":"2024-09-21T23:25:28.165264+0000","last_unstale":"2024-09-21T23:26:04.331866+0000","last_undegraded":"2024-09-21T23:26:04.331866+0000","last_fullsized":"2024-09-21T23:26:04.331866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:39:53.806934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053117699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-21T23:26:04.331306+0000","last_change":"2024-09-21T23:25:30.263565+0000","last_active":"2024-09-21T23:26:04.331306+0000","last_peered":"2024-09-21T23:26:04.331306+0000","last_clean":"2024-09-21T23:26:04.331306+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:26:04.331306+0000","last_undegraded":"2024-09-21T23:26:04.331306+0000","last_fullsized":"2024-09-21T23:26:04.331306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:46:37.545348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042085499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-21T23:26:04.331747+0000","last_change":"2024-09-21T23:25:30.189121+0000","last_active":"2024-09-21T23:26:04.331747+0000","last_peered":"2024-09-21T23:26:04.331747+0000","last_clean":"2024-09-21T23:26:04.331747+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:26:04.331747+0000","last_undegraded":"2024-09-21T23:26:04.331747+0000","last_fullsized":"2024-09-21T23:26:04.331747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:00:22.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054857199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":54,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-21T23:26:05.499457+0000","last_change":"2024-09-21T23:25:30.186984+0000","last_active":"2024-09-21T23:26:05.499457+0000","last_peered":"2024-09-21T23:26:05.499457+0000","last_clean":"2024-09-21T23:26:05.499457+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:26:05.499457+0000","last_undegraded":"2024-09-21T23:26:05.499457+0000","last_fullsized":"2024-09-21T23:26:05.499457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:01:38.740017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095847300000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-21T23:26:04.334264+0000","last_change":"2024-09-21T23:25:30.263343+0000","last_active":"2024-09-21T23:26:04.334264+0000","last_peered":"2024-09-21T23:26:04.334264+0000","last_clean":"2024-09-21T23:26:04.334264+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:26:04.334264+0000","last_undegraded":"2024-09-21T23:26:04.334264+0000","last_fullsized":"2024-09-21T23:26:04.334264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:50:16.699815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044148899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-21T23:26:04.331306+0000","last_change":"2024-09-21T23:25:30.263501+0000","last_active":"2024-09-21T23:26:04.331306+0000","last_peered":"2024-09-21T23:26:04.331306+0000","last_clean":"2024-09-21T23:26:04.331306+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:26:04.331306+0000","last_undegraded":"2024-09-21T23:26:04.331306+0000","last_fullsized":"2024-09-21T23:26:04.331306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.155893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036082099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":53,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-21T23:26:05.499348+0000","last_change":"2024-09-21T23:25:30.186477+0000","last_active":"2024-09-21T23:26:05.499348+0000","last_peered":"2024-09-21T23:26:05.499348+0000","last_clean":"2024-09-21T23:26:05.499348+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:26:05.499348+0000","last_undegraded":"2024-09-21T23:26:05.499348+0000","last_fullsized":"2024-09-21T23:26:05.499348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:32:47.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044681799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":174,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-21T23:26:01.321646+0000","last_change":"2024-09-21T23:26:01.315848+0000","last_active":"2024-09-21T23:26:01.321646+0000","last_peered":"2024-09-21T23:26:01.321646+0000","last_clean":"2024-09-21T23:26:01.321646+0000","last_became_active":"2024-09-21T23:26:01.315355+0000","last_became_peered":"2024-09-21T23:26:01.315355+0000","last_unstale":"2024-09-21T23:26:01.321646+0000","last_undegraded":"2024-09-21T23:26:01.321646+0000","last_fullsized":"2024-09-21T23:26:01.321646+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:39:32.215042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":2361472,"data_compressed":37034,"data_compressed_allocated":147456,"data_compressed_original":2326528,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542157,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":417792,"data_stored":164101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509454,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":409600,"data_stored":158591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542156,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":458752,"data_stored":754469,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":417792,"data_stored":164101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509454,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":421888,"data_stored":164120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":417792,"data_stored":164101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":405504,"data_stored":158572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":446464,"data_stored":748940,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:26:09.794 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:09.794 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:09.806 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:09.866 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:09.866+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:09.954 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:09.955 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:09.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:10.007 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:10.007+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:10.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:10.067+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:10.113 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:10.113 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:10.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:10.208 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:10.208+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:10.270 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:10.269+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:10.275 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:10.275 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:10.287 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:10.348 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-09-21T23:26:10.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:10.370+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:10.409 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:10.409 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:10.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:10.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:10.470+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:10.526 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:10.526 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:10.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:10.572 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:10.572+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:10.641 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:10.642 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:10.650 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:10.694 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:10.694+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:10.758 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:10.758 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:10.767 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:10.808 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:10.808+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:10.875 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:10.875+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:10.914 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:10.914 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:10.923 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:11.007 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:11.007+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:11.071 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:11.071 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:11.077 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:11.077+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:11.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:11.178 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:11.177+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:11.227 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:11.227 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:11.236 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:11.279 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:11.279+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:11.343 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:11.343 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:11.351 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:11.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:26:11.413 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:11.413+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:11.480 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:11.479+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:11.498 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:11.498 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:11.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:11.524 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:11.524 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:11.557 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:11.558 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:11.574 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:11.574 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:11.580 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:11.579+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:11.605 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:11.605 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:11.615 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:11.615 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:11.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:11.628 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:11.628 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:11.660 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:11.660 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:11.676 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:11.681 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:11.681+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:11.693 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:11.693 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:11.733 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:11.733 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:11.733 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:11.733 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:11.733 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:11.742 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:11.781 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:11.781+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:11.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:11.882+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:11.892 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:11.892 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:11.892 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:11.892 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:11.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.892 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.723501837999997, 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090066050000000009, 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 37 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7099999999999998e-07 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167009+0000", 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.92e-06 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167013+0000", 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167017+0000", 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7280000000000001e-06 2024-09-21T23:26:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170778+0000", 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037612399999999999 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170831+0000", 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2694000000000001e-05 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170970+0000", 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000138966 2024-09-21T23:26:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.171555+0000", 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058538399999999999 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175872+0000", 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043172389999999996 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175969+0000", 2024-09-21T23:26:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6844999999999999e-05 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.176012+0000", 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3356999999999998e-05 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.697397002000002, 2024-09-21T23:26:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069023069999999999, 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0999999999999999e-07 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193115+0000", 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5700000000000001e-06 2024-09-21T23:26:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193129+0000", 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4124000000000001e-05 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193133+0000", 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0550000000000001e-06 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194144+0000", 2024-09-21T23:26:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001010735 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194231+0000", 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7665999999999997e-05 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194357+0000", 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125353 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194844+0000", 2024-09-21T23:26:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000487305 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199964+0000", 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051204859999999996 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199992+0000", 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7902000000000002e-05 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.200013+0000", 2024-09-21T23:26:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0601000000000001e-05 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.675747928, 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0067494620000000003, 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 73 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4900000000000001e-07 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214818+0000", 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7754000000000003e-05 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214842+0000", 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4552000000000001e-05 2024-09-21T23:26:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214846+0000", 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9129999999999996e-06 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215813+0000", 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00096695899999999998 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215904+0000", 2024-09-21T23:26:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0328000000000007e-05 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216020+0000", 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115838 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217018+0000", 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099836200000000003 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221461+0000", 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044435969999999997 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221487+0000", 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5806e-05 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221509+0000", 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1903999999999999e-05 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.675653224999998, 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068139230000000004, 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 74 2024-09-21T23:26:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214855+0000", 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-07 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214859+0000", 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1389999999999997e-06 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214871+0000", 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1698999999999999e-05 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214887+0000", 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6061000000000001e-05 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216359+0000", 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014720779999999999 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216439+0000", 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0191999999999994e-05 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216579+0000", 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014029099999999999 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217129+0000", 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054932099999999997 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221559+0000", 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044299710000000004 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221627+0000", 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8631000000000001e-05 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221668+0000", 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0988000000000001e-05 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.658977039, 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0077112509999999997, 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 86 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5199999999999998e-07 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231533+0000", 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3879999999999998e-06 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231537+0000", 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0010000000000003e-06 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231540+0000", 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0489999999999999e-06 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233890+0000", 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023500249999999999 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233937+0000", 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6400999999999998e-05 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234015+0000", 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8343000000000005e-05 2024-09-21T23:26:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235675+0000", 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001660124 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239138+0000", 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034624109999999999 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239200+0000", 2024-09-21T23:26:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2698000000000002e-05 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239242+0000", 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1458999999999997e-05 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.65896077, 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010200097999999999, 2024-09-21T23:26:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2700000000000001e-07 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231588+0000", 2024-09-21T23:26:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0985000000000001e-05 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231593+0000", 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1059999999999999e-06 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231606+0000", 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2343e-05 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234406+0000", 2024-09-21T23:26:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027999850000000001 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234489+0000", 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3051000000000002e-05 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234631+0000", 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285799999999999 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235679+0000", 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001047517 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241644+0000", 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059652129999999996 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241704+0000", 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9732999999999999e-05 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241747+0000", 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-05 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.658894930999999, 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010246534, 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 88 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0199999999999998e-07 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231615+0000", 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3439999999999999e-06 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231619+0000", 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0219999999999998e-06 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231622+0000", 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6989999999999999e-06 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235337+0000", 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037150030000000001 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235385+0000", 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7861999999999997e-05 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235458+0000", 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2596999999999996e-05 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235978+0000", 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052013600000000004 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241799+0000", 2024-09-21T23:26:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058215289999999998 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241836+0000", 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6282999999999998e-05 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241859+0000", 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3757e-05 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.501503747999998, 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072206969999999999, 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 112 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.72e-07 2024-09-21T23:26:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389069+0000", 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4875000000000005e-05 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389079+0000", 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0135e-05 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389085+0000", 2024-09-21T23:26:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394006+0000", 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049207920000000002 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394052+0000", 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5810999999999999e-05 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394139+0000", 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7130000000000001e-05 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395407+0000", 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012681260000000001 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396139+0000", 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000731703 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396194+0000", 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5309999999999997e-05 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396225+0000", 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0589000000000001e-05 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.501396790999998, 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086923490000000003, 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 113 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389116+0000", 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.245e-06 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389126+0000", 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0295000000000001e-05 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389132+0000", 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0889999999999996e-06 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394548+0000", 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005415876 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394642+0000", 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4531999999999995e-05 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394714+0000", 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1753000000000002e-05 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396233+0000", 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015185299999999999 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397749+0000", 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015157770000000001 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397779+0000", 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0561000000000001e-05 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397803+0000", 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4116000000000001e-05 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.501364721999998, 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.01139824, 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 114 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.01e-07 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389146+0000", 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5239999999999999e-06 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389150+0000", 2024-09-21T23:26:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3989999999999997e-06 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389153+0000", 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.038e-06 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395026+0000", 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058731720000000003 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395071+0000", 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4388999999999998e-05 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395141+0000", 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0053000000000002e-05 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397196+0000", 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020555899999999999 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400494+0000", 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032972679999999999 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400520+0000", 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6423000000000001e-05 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400541+0000", 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0982999999999999e-05 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.501346042999998, 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011537824, 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 115 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4900000000000001e-07 2024-09-21T23:26:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389197+0000", 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5204999999999999e-05 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389203+0000", 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0979999999999999e-06 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389207+0000", 2024-09-21T23:26:11.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6600000000000001e-06 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395443+0000", 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062358099999999996 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395525+0000", 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1855999999999997e-05 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395669+0000", 2024-09-21T23:26:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014472399999999999 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397204+0000", 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001534817 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400599+0000", 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033944499999999998 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400655+0000", 2024-09-21T23:26:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6647000000000002e-05 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400699+0000", 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4208000000000001e-05 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.501294590999997, 2024-09-21T23:26:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011571580999999999, 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 116 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3099999999999999e-07 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389216+0000", 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4260000000000002e-06 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389223+0000", 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389225+0000", 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7760000000000002e-06 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396256+0000", 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070309020000000003 2024-09-21T23:26:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396348+0000", 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1780000000000006e-05 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396481+0000", 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000133101 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397208+0000", 2024-09-21T23:26:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072716199999999995 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400737+0000", 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035290220000000001 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400763+0000", 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5219999999999999e-05 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400785+0000", 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2045e-05 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.501277907999999, 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011687858000000001, 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 117 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3699999999999999e-07 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389232+0000", 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1789999999999998e-06 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389238+0000", 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209999999999999e-06 2024-09-21T23:26:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389241+0000", 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.557e-06 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396792+0000", 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075506899999999997 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396838+0000", 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5639000000000001e-05 2024-09-21T23:26:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396958+0000", 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119892 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397458+0000", 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049987600000000001 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400824+0000", 2024-09-21T23:26:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033662560000000002 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400880+0000", 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5779999999999998e-05 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400918+0000", 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8031000000000002e-05 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.489259433999997, 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006886924, 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 120 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401249+0000", 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.13e-07 2024-09-21T23:26:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401286+0000", 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6773999999999997e-05 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401295+0000", 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9039999999999998e-06 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401299+0000", 2024-09-21T23:26:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3250000000000001e-06 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402183+0000", 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000883758 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402264+0000", 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1080000000000003e-05 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402401+0000", 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013719700000000001 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403430+0000", 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010290119999999999 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408045+0000", 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046146659999999999 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408095+0000", 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0898e-05 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408135+0000", 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9697000000000003e-05 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.489200959999998, 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069210920000000002, 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 121 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3200000000000001e-07 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401341+0000", 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4236999999999998e-05 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401346+0000", 2024-09-21T23:26:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.04e-06 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401350+0000", 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6399999999999999e-06 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402733+0000", 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013825790000000001 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402816+0000", 2024-09-21T23:26:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3461000000000006e-05 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402941+0000", 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012518300000000001 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403533+0000", 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000592071 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408174+0000", 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046404209999999996 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408204+0000", 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0147000000000001e-05 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408228+0000", 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3981e-05 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.053058063000002, 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064924400000000004, 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 139 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0800000000000001e-07 2024-09-21T23:26:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837452+0000", 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7209999999999999e-06 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837455+0000", 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2279999999999998e-06 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837458+0000", 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8200000000000001e-06 2024-09-21T23:26:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842187+0000", 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047298460000000002 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842233+0000", 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6056000000000001e-05 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842304+0000", 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0862e-05 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843291+0000", 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00098632100000000007 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843894+0000", 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060369199999999997 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843922+0000", 2024-09-21T23:26:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7429000000000001e-05 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843942+0000", 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0256999999999999e-05 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.053046119999998, 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066155550000000004, 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 140 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1e-07 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837517+0000", 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5034000000000002e-05 2024-09-21T23:26:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837522+0000", 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8520000000000003e-06 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837525+0000", 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4139999999999998e-06 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842562+0000", 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050366279999999996 2024-09-21T23:26:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842615+0000", 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3675999999999999e-05 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842715+0000", 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9990999999999999e-05 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843873+0000", 2024-09-21T23:26:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011573 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844026+0000", 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015368900000000001 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844058+0000", 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1449999999999999e-05 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844077+0000", 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9310999999999999e-05 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.052973926999996, 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068324850000000001, 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 141 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.22e-07 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837539+0000", 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.023e-06 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837543+0000", 2024-09-21T23:26:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6490000000000001e-06 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837546+0000", 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.382e-06 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843479+0000", 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059328189999999998 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843519+0000", 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0071000000000003e-05 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843600+0000", 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1376000000000002e-05 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844137+0000", 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053640299999999997 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844319+0000", 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000181893 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844346+0000", 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.781e-05 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844366+0000", 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9737000000000001e-05 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 49.437640232, 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006594145, 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 158 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452868+0000", 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9500000000000003e-07 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452938+0000", 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9598999999999996e-05 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452950+0000", 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2971e-05 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452960+0000", 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.448e-06 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456217+0000", 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003257378 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456557+0000", 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033989399999999998 2024-09-21T23:26:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456703+0000", 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000145599 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459339+0000", 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026365529999999998 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459386+0000", 2024-09-21T23:26:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6640000000000001e-05 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459428+0000", 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2148999999999999e-05 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459462+0000", 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3419e-05 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 49.437535857999997, 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0065672760000000004, 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 159 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5400000000000001e-07 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452977+0000", 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.301e-06 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452984+0000", 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1840000000000002e-06 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452989+0000", 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5219999999999998e-06 2024-09-21T23:26:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457324+0000", 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043348780000000003 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457416+0000", 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1824999999999996e-05 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457574+0000", 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015810999999999999 2024-09-21T23:26:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459376+0000", 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001801649 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459467+0000", 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1367999999999997e-05 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459507+0000", 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9576000000000002e-05 2024-09-21T23:26:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459539+0000", 2024-09-21T23:26:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2308999999999998e-05 2024-09-21T23:26:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:12.008 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:12.010 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:12.010+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:12.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:12.110 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:12.110+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:12.139 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:12.140 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:12.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:12.232 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:12.232+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:12.297 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:12.297 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:12.297 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:12.297 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:12.297 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:12.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:12.346 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:12.346+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:12.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T23:26:12.412 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:12.412+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:12.461 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.323073055999998, 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036127939999999999, 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3500000000000002e-07 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136608+0000", 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.182e-06 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136618+0000", 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9199999999999999e-06 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136628+0000", 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4390000000000006e-06 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136692+0000", 2024-09-21T23:26:12.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3979000000000005e-05 2024-09-21T23:26:12.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136716+0000", 2024-09-21T23:26:12.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4391000000000001e-05 2024-09-21T23:26:12.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140171+0000", 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034554910000000002 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140214+0000", 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2057000000000001e-05 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'18)", 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.288554892999997, 2024-09-21T23:26:12.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043957730000000004, 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171120+0000", 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6700000000000003e-07 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171125+0000", 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.451e-06 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171138+0000", 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2541000000000001e-05 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171143+0000", 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4659999999999998e-06 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171187+0000", 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4014000000000002e-05 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.469 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171200+0000", 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3108999999999999e-05 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175479+0000", 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042784570000000003 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175515+0000", 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5967999999999999e-05 2024-09-21T23:26:12.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.471 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.471 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.471 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:58 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-21T23:26:12.471 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:12.471 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.265802596999997, 2024-09-21T23:26:12.471 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003032851, 2024-09-21T23:26:12.471 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.471 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.471 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193872+0000", 2024-09-21T23:26:12.472 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9400000000000004e-07 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193952+0000", 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9791999999999998e-05 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193962+0000", 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0034000000000001e-05 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193973+0000", 2024-09-21T23:26:12.473 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0733e-05 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194015+0000", 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2911999999999998e-05 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194031+0000", 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5082999999999999e-05 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196869+0000", 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002838808 2024-09-21T23:26:12.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196904+0000", 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4694999999999999e-05 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:59 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.265216930999998, 2024-09-21T23:26:12.475 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053675709999999998, 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7100000000000002e-07 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194462+0000", 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.668e-06 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194469+0000", 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1670000000000002e-06 2024-09-21T23:26:12.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194475+0000", 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-06 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194513+0000", 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7391999999999997e-05 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194531+0000", 2024-09-21T23:26:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7870999999999999e-05 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199791+0000", 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052604119999999999 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199824+0000", 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3340000000000003e-05 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:60 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'38, mlcod=16'36)", 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.259136236000003, 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003614562, 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200539+0000", 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5799999999999999e-06 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200551+0000", 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1816e-05 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200578+0000", 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6772999999999999e-05 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200590+0000", 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2303000000000001e-05 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200715+0000", 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012523499999999999 2024-09-21T23:26:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200731+0000", 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5690000000000001e-05 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204121+0000", 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003390363 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204152+0000", 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0802000000000001e-05 2024-09-21T23:26:12.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:70 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'39)", 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.249080581000001, 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033896579999999998, 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210594+0000", 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1099999999999995e-07 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210600+0000", 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4389999999999997e-06 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210613+0000", 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.243e-05 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210618+0000", 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0939999999999998e-06 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210843+0000", 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022493200000000001 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210858+0000", 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5109000000000001e-05 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213938+0000", 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030796500000000002 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213983+0000", 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5293000000000002e-05 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:72 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'46, mlcod=16'43)", 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.244135804999999, 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029102279999999999, 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.488 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215539+0000", 2024-09-21T23:26:12.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4199999999999995e-07 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215589+0000", 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0488999999999997e-05 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215598+0000", 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2830000000000007e-06 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215610+0000", 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1134999999999999e-05 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215657+0000", 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7658999999999999e-05 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215679+0000", 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1526999999999999e-05 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218410+0000", 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027316939999999998 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218448+0000", 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7799000000000002e-05 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:73 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'47, mlcod=16'43)", 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:12.492 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.243526457999998, 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051882020000000003, 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216148+0000", 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7100000000000001e-07 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216218+0000", 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0313000000000003e-05 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216227+0000", 2024-09-21T23:26:12.494 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9959999999999999e-06 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216232+0000", 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5340000000000001e-06 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216267+0000", 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4739000000000001e-05 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.495 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216281+0000", 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4345000000000001e-05 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221290+0000", 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050088989999999998 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221336+0000", 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5704999999999997e-05 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:74 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'48, mlcod=16'43)", 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.243000514999999, 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046922939999999996, 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216674+0000", 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9900000000000001e-07 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216679+0000", 2024-09-21T23:26:12.498 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.502e-06 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216687+0000", 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1790000000000008e-06 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216691+0000", 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7230000000000002e-06 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216737+0000", 2024-09-21T23:26:12.499 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6087000000000001e-05 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216758+0000", 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0434000000000001e-05 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221338+0000", 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045804569999999996 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.500 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221366+0000", 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7413e-05 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:86 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'56, mlcod=16'50)", 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.225578581000001, 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049055269999999998, 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.501 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234096+0000", 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234099+0000", 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3890000000000001e-06 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234107+0000", 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2349999999999999e-06 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234112+0000", 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0629999999999999e-06 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234143+0000", 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1131999999999999e-05 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234155+0000", 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2140000000000001e-05 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.238972+0000", 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048167770000000004 2024-09-21T23:26:12.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.239001+0000", 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9164000000000001e-05 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:87 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'57, mlcod=16'52)", 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.224899206000003, 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067179910000000004, 2024-09-21T23:26:12.505 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.506 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0299999999999998e-07 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234813+0000", 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7531999999999999e-05 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234826+0000", 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2617e-05 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.507 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234831+0000", 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1229999999999999e-06 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234869+0000", 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8513999999999998e-05 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234881+0000", 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2148e-05 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241465+0000", 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065837329999999996 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241493+0000", 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7521e-05 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:88 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-21T23:26:12.509 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.224101357999999, 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059344530000000001, 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.510 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235573+0000", 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0100000000000004e-07 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235578+0000", 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1590000000000001e-06 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.511 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235590+0000", 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1229e-05 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235595+0000", 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0889999999999999e-06 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235943+0000", 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000348227 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.512 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235957+0000", 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4419999999999999e-05 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241494+0000", 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055371070000000003 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241507+0000", 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2520999999999999e-05 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:113 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'75, mlcod=16'71)", 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.064870079999999, 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028543489999999999, 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8199999999999999e-07 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394809+0000", 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394821+0000", 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2024999999999999e-05 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394827+0000", 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0020000000000003e-06 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394862+0000", 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4811000000000003e-05 2024-09-21T23:26:12.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394873+0000", 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.058e-05 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397631+0000", 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027581060000000002 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397658+0000", 2024-09-21T23:26:12.517 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7073999999999999e-05 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'71)", 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.064458666999997, 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005160085, 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.518 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395216+0000", 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4999999999999998e-07 2024-09-21T23:26:12.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395221+0000", 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2290000000000002e-06 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395229+0000", 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2369999999999992e-06 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.520 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395233+0000", 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1840000000000001e-06 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395260+0000", 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.728e-05 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395272+0000", 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1270000000000001e-05 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.521 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400341+0000", 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050691809999999999 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400375+0000", 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4254000000000001e-05 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'72)", 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:12.522 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.063910145999998, 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046274580000000001, 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395829+0000", 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4618999999999999e-05 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395836+0000", 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9090000000000003e-06 2024-09-21T23:26:12.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395839+0000", 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.371e-06 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395868+0000", 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9328000000000001e-05 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395880+0000", 2024-09-21T23:26:12.525 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1501000000000001e-05 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400376+0000", 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044966340000000002 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400391+0000", 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4676e-05 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:116 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'78, mlcod=16'73)", 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.063105598999996, 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038400970000000002, 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.527 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396569+0000", 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1100000000000001e-07 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396573+0000", 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5569999999999997e-06 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.528 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396579+0000", 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1120000000000001e-06 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396587+0000", 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396630+0000", 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2991000000000002e-05 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.529 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396642+0000", 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1481000000000001e-05 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400392+0000", 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037507830000000002 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400408+0000", 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5863e-05 2024-09-21T23:26:12.530 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:117 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'79, mlcod=16'73)", 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.062603256999999, 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033512120000000001, 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.531 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7e-07 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.532 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397075+0000", 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7809999999999999e-06 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397082+0000", 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397085+0000", 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7139999999999999e-06 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397130+0000", 2024-09-21T23:26:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.498e-05 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397142+0000", 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1838000000000001e-05 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400409+0000", 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032666349999999999 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400422+0000", 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2866e-05 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.057779584000002, 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034761240000000001, 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401895+0000", 2024-09-21T23:26:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7400000000000003e-07 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401963+0000", 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8701000000000005e-05 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401972+0000", 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9530000000000005e-06 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401977+0000", 2024-09-21T23:26:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1579999999999998e-06 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402009+0000", 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2512000000000003e-05 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402022+0000", 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3220999999999999e-05 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.538 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405337+0000", 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033150300000000001 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405370+0000", 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2975000000000002e-05 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.057173513999999, 2024-09-21T23:26:12.539 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054168369999999999, 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402501+0000", 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2699999999999999e-07 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402589+0000", 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899000000000007e-05 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402599+0000", 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0312000000000001e-05 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402605+0000", 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9689999999999999e-06 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402648+0000", 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3210000000000001e-05 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402664+0000", 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5347999999999999e-05 2024-09-21T23:26:12.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407887+0000", 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052237919999999997 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407917+0000", 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9779999999999999e-05 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.543 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 51.056644132999999, 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049049180000000003, 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4099999999999999e-07 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403098+0000", 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7464000000000003e-05 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403106+0000", 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6629999999999999e-06 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403111+0000", 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403137+0000", 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6349000000000001e-05 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403148+0000", 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.077e-05 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407920+0000", 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047716590000000001 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407935+0000", 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4783999999999999e-05 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.547 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.548 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.548 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.548 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.548 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:12.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:12.549 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:12.548+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:12.650 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:12.649+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:12.709 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:12.709 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:12.709 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:12.709 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:12.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:12.751 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:12.750+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:12.829 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:12.830 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:12.830 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:12.830 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:12.830 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:12.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:12.882 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:12.882+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:12.946 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 43.775955103999998, 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003527526, 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170743+0000", 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.08e-07 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170750+0000", 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800000000000001e-06 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170767+0000", 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6923999999999999e-05 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170788+0000", 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0446e-05 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170849+0000", 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1409999999999996e-05 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170881+0000", 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2075e-05 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174218+0000", 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033371070000000002 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174270+0000", 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1375999999999997e-05 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 42.759965610000002, 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011783570000000001, 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7999999999999996e-07 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186750+0000", 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7444000000000001e-05 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186834+0000", 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3974999999999995e-05 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:12.953 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186876+0000", 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2796000000000002e-05 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186999+0000", 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012303800000000001 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187036+0000", 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6897000000000003e-05 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187869+0000", 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00083215199999999996 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187910+0000", 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1575000000000001e-05 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:12.955 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:12.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:12.957 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:12.956+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:13.057 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:13.056+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:13.107 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:13.107 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:13.108 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:13.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:13.161 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:13.161+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:13.225 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:13.225 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:13.225 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:13.225 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:13.225 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:13.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:13.258 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:13.257+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:13.359 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:13.358+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:13.359 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-09-21T23:26:13.407 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:13.407 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:13.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:1 2.3 e19/18 2:c4c92e5a:::rbd_trash:head v 19'1, mlcod=0'0)", 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 45.239842582000001, 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039243489999999997, 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:13.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2999999999999999e-07 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167667+0000", 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7169999999999997e-06 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167679+0000", 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.276e-05 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167688+0000", 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8380000000000008e-06 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167739+0000", 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0964999999999997e-05 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167762+0000", 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2221999999999998e-05 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171515+0000", 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037531119999999999 2024-09-21T23:26:13.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:13.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171582+0000", 2024-09-21T23:26:13.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7504999999999998e-05 2024-09-21T23:26:13.412 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:13.412 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:13.412 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:13.412 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:13.412 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:13.412 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:13.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:13.460 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:13.459+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:13.561 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:13.560+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:13.631 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:13.631 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:13.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:13.662 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:13.661+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:13.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:26:13.759 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:13.759 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:13.762 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:13.761+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:13.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:13.850 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:13.850 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:13.862 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:13.862+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:13.873 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:13.873 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 14.581365337999999, 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00081488299999999997, 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301851+0000", 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7899999999999998e-07 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302047+0000", 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019639200000000001 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302064+0000", 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7380999999999998e-05 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302080+0000", 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5237e-05 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302153+0000", 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3922000000000001e-05 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302162+0000", 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8170000000000005e-06 2024-09-21T23:26:13.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302665+0000", 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050245499999999996 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 14.57858212, 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00068581899999999995, 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:13.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:13.888 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304634+0000", 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5499999999999998e-07 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304796+0000", 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016237700000000001 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304813+0000", 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.732e-05 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.889 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304825+0000", 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1302e-05 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304896+0000", 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1729000000000004e-05 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304905+0000", 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8359999999999998e-06 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.305319+0000", 2024-09-21T23:26:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00041360000000000002 2024-09-21T23:26:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:13.891 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:13.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:13.905 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:13.937 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:13.937 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:13.954 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:13.954 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:13.963 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:13.963+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:13.971 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:13.972 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:13.988 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:13.989 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:14.005 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:14.049 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:14.050 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:14.050 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:14.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:14.094 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:14.093+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:14.158 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:14.158 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:14.164 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:14.164+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:14.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:14.266 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:14.265+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:14.313 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:14.313 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:14.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:14.367 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:14.367+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:14.429 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:14.429 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:14.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:14.468 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:14.468+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:14.545 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:14.545 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:14.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:14.597 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:14.596+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:14.661 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:14.661 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:14.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:14.671 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:14.668+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:14.770 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:14.769+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:14.789 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:26:14.790 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:26:14.790 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T23:26:14.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T23:26:14.818 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:14.818 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:14.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:14.871 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:14.870+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:14.973 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:14.972+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:14.975 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:14.975 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:14.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:15.073 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:15.073+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:15.091 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:15.091 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:15.100 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:15.132 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:26:15.132 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 41, 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "d51cc561-ddb8-41ef-b61c-7c6d698e4dd0", 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-09-21T23:25:09.056809+0000", 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-09-21T23:26:13.353658+0000", 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-09-21T23:25:20.126996+0000", 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-09-21T23:25:11.226810+0000", 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5801984, 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "noscrub", 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-09-21T23:26:15.133 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 5, 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 2, 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 8, 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "luminous", 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-09-21T23:26:15.134 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:19.818212+0000", 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:15.135 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 1, 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 1, 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 1, 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 1, 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 1, 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:15.136 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "17", 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-09-21T23:26:15.137 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:15.138 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:15.139 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:15.140 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 6.9000000953674316, 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 6.9000000953674316, 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.28999999165534973, 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 2, 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "rbd", 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:26.968096+0000", 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 8193, 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T23:26:15.141 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 8, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 8, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 8, 2024-09-21T23:26:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 8, 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 8, 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "21", 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 2, 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 21, 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:15.144 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:15.145 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd": {} 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 1.75, 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 1.75, 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 1, 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 1.75, 2024-09-21T23:26:15.146 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 1.75, 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af", 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:15.147 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 35, 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6817/2339966024", 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6819/2339966024", 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6823/2339966024", 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6821/2339966024", 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "88ee06a3-9a6a-41e8-9327-a7a61304c03d", 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:15.155 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6825/2889649822", 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6827/2889649822", 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6831/2889649822", 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6829/2889649822", 2024-09-21T23:26:15.157 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05", 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:15.158 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-09-21T23:26:15.159 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:15.160 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-09-21T23:26:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-09-21T23:26:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6809/864917651", 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6811/864917651", 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6815/864917651", 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6813/864917651", 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "562e8c30-0622-4cb3-b5d1-f15b42a444bc", 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 36, 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6801/1713632981", 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6803/1713632981", 2024-09-21T23:26:15.168 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6807/1713632981", 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6805/1713632981", 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7030ff68-4556-4ad5-abc9-be758f1caa85", 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:15.169 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6808", 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:15.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6809", 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6810", 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6811", 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6814", 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6815", 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6812", 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6813", 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6809/2660753894", 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6811/2660753894", 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6815/2660753894", 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6813/2660753894", 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7169a163-f067-4385-b734-f58a3ba9e89e", 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:15.175 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6824", 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6825", 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.176 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6826", 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6827", 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.177 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6830", 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6831", 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.178 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6828", 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6829", 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.179 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6825/2320687649", 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6827/2320687649", 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6831/2320687649", 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6829/2320687649", 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e511295d-2177-4dcb-a74b-b16b92534df6", 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:15.180 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6800", 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.181 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6801", 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6802", 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6803", 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6806", 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.183 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6807", 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6804", 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6805", 2024-09-21T23:26:15.184 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6801/3490482162", 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6803/3490482162", 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3490482162", 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6805/3490482162", 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.185 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "544f8a91-ce82-4855-85eb-ea814eb3d058", 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:15.186 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6816", 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6817", 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:15.187 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6818", 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6819", 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6822", 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6823", 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6820", 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6821", 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6817/1315554966", 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6819/1315554966", 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6823/1315554966", 2024-09-21T23:26:15.190 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6821/1315554966", 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:15.191 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.514872+0000", 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.912129+0000", 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:15.192 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.745154+0000", 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:15.193 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:16.758871+0000", 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.287462+0000", 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:15.194 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:18.031636+0000", 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:15.195 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.055855+0000", 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.524609+0000", 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:15.196 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [ 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: 3, 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: 4 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [ 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "2.0", 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: "mappings": [ 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:15.197 INFO:teuthology.orchestra.run.smithi031.stdout: "from": 1, 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: "to": 7 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [], 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": {}, 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-09-21T23:26:15.198 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-09-21T23:26:15.199 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-09-21T23:26:15.200 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-09-21T23:26:15.200 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-09-21T23:26:15.200 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:15.200 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:15.202 INFO:tasks.thrashosds.thrasher:j is {'epoch': 41, 'fsid': 'd51cc561-ddb8-41ef-b61c-7c6d698e4dd0', 'created': '2024-09-21T23:25:09.056809+0000', 'modified': '2024-09-21T23:26:13.353658+0000', 'last_up_change': '2024-09-21T23:25:20.126996+0000', 'last_in_change': '2024-09-21T23:25:11.226810+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T23:25:19.818212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T23:25:26.968096+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 2339966024}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 2339966024}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 2339966024}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 2339966024}]}, 'public_addr': '172.21.15.31:6817/2339966024', 'cluster_addr': '172.21.15.31:6819/2339966024', 'heartbeat_back_addr': '172.21.15.31:6823/2339966024', 'heartbeat_front_addr': '172.21.15.31:6821/2339966024', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '88ee06a3-9a6a-41e8-9327-a7a61304c03d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 2889649822}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2889649822}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2889649822}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2889649822}]}, 'public_addr': '172.21.15.31:6825/2889649822', 'cluster_addr': '172.21.15.31:6827/2889649822', 'heartbeat_back_addr': '172.21.15.31:6831/2889649822', 'heartbeat_front_addr': '172.21.15.31:6829/2889649822', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 864917651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 864917651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 864917651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 864917651}]}, 'public_addr': '172.21.15.31:6809/864917651', 'cluster_addr': '172.21.15.31:6811/864917651', 'heartbeat_back_addr': '172.21.15.31:6815/864917651', 'heartbeat_front_addr': '172.21.15.31:6813/864917651', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '562e8c30-0622-4cb3-b5d1-f15b42a444bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 36, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6800', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6801', 'nonce': 1713632981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1713632981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1713632981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1713632981}]}, 'public_addr': '172.21.15.31:6801/1713632981', 'cluster_addr': '172.21.15.31:6803/1713632981', 'heartbeat_back_addr': '172.21.15.31:6807/1713632981', 'heartbeat_front_addr': '172.21.15.31:6805/1713632981', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7030ff68-4556-4ad5-abc9-be758f1caa85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6809', 'nonce': 2660753894}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6811', 'nonce': 2660753894}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6815', 'nonce': 2660753894}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6813', 'nonce': 2660753894}]}, 'public_addr': '172.21.15.60:6809/2660753894', 'cluster_addr': '172.21.15.60:6811/2660753894', 'heartbeat_back_addr': '172.21.15.60:6815/2660753894', 'heartbeat_front_addr': '172.21.15.60:6813/2660753894', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7169a163-f067-4385-b734-f58a3ba9e89e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6824', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6825', 'nonce': 2320687649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6826', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6827', 'nonce': 2320687649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6830', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6831', 'nonce': 2320687649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6828', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6829', 'nonce': 2320687649}]}, 'public_addr': '172.21.15.60:6825/2320687649', 'cluster_addr': '172.21.15.60:6827/2320687649', 'heartbeat_back_addr': '172.21.15.60:6831/2320687649', 'heartbeat_front_addr': '172.21.15.60:6829/2320687649', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e511295d-2177-4dcb-a74b-b16b92534df6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6801', 'nonce': 3490482162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6803', 'nonce': 3490482162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6807', 'nonce': 3490482162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6805', 'nonce': 3490482162}]}, 'public_addr': '172.21.15.60:6801/3490482162', 'cluster_addr': '172.21.15.60:6803/3490482162', 'heartbeat_back_addr': '172.21.15.60:6807/3490482162', 'heartbeat_front_addr': '172.21.15.60:6805/3490482162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '544f8a91-ce82-4855-85eb-ea814eb3d058', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6816', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6817', 'nonce': 1315554966}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6818', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6819', 'nonce': 1315554966}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6822', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6823', 'nonce': 1315554966}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6820', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6821', 'nonce': 1315554966}]}, 'public_addr': '172.21.15.60:6817/1315554966', 'cluster_addr': '172.21.15.60:6819/1315554966', 'heartbeat_back_addr': '172.21.15.60:6823/1315554966', 'heartbeat_front_addr': '172.21.15.60:6821/1315554966', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.514872+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.912129+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.745154+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:16.758871+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.287462+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:18.031636+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.055855+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.524609+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [3, 4]}], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 1, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T23:26:15.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:26:15.202 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:15.200+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:15.207 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:15.207 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:15.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:15.302 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:15.301+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:15.364 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:15.364 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:15.374 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:15.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T23:26:15.418 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:15.418+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:15.504 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:15.502+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:15.521 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:15.521 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:15.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:15.605 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:15.604+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:15.706 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:15.705+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:15.710 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:15.711 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:15.719 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:26:15.720 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:26:15.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:15.738 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-21T23:26:13.783749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236348,"kb_used_data":3484,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738372,"statfs":{"total":773094113280,"available":772852092928,"internally_reserved":0,"allocated":3567616,"data_stored":2554082,"data_compressed":19550,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.442895"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":55,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-21T23:26:07.515380+0000","last_change":"2024-09-21T23:25:30.180125+0000","last_active":"2024-09-21T23:26:07.515380+0000","last_peered":"2024-09-21T23:26:07.515380+0000","last_clean":"2024-09-21T23:26:07.515380+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:26:07.515380+0000","last_undegraded":"2024-09-21T23:26:07.515380+0000","last_fullsized":"2024-09-21T23:26:07.515380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:17.515296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033759699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":56,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-21T23:26:10.567191+0000","last_change":"2024-09-21T23:25:30.189119+0000","last_active":"2024-09-21T23:26:10.567191+0000","last_peered":"2024-09-21T23:26:10.567191+0000","last_clean":"2024-09-21T23:26:10.567191+0000","last_became_active":"2024-09-21T23:25:28.165264+0000","last_became_peered":"2024-09-21T23:25:28.165264+0000","last_unstale":"2024-09-21T23:26:10.567191+0000","last_undegraded":"2024-09-21T23:26:10.567191+0000","last_fullsized":"2024-09-21T23:26:10.567191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:39:53.806934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053117699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":55,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-21T23:26:07.341930+0000","last_change":"2024-09-21T23:25:30.263565+0000","last_active":"2024-09-21T23:26:07.341930+0000","last_peered":"2024-09-21T23:26:07.341930+0000","last_clean":"2024-09-21T23:26:07.341930+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:26:07.341930+0000","last_undegraded":"2024-09-21T23:26:07.341930+0000","last_fullsized":"2024-09-21T23:26:07.341930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:46:37.545348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042085499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":56,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-21T23:26:10.567170+0000","last_change":"2024-09-21T23:25:30.189121+0000","last_active":"2024-09-21T23:26:10.567170+0000","last_peered":"2024-09-21T23:26:10.567170+0000","last_clean":"2024-09-21T23:26:10.567170+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:26:10.567170+0000","last_undegraded":"2024-09-21T23:26:10.567170+0000","last_fullsized":"2024-09-21T23:26:10.567170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:00:22.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054857199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":56,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-21T23:26:07.515275+0000","last_change":"2024-09-21T23:25:30.186984+0000","last_active":"2024-09-21T23:26:07.515275+0000","last_peered":"2024-09-21T23:26:07.515275+0000","last_clean":"2024-09-21T23:26:07.515275+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:26:07.515275+0000","last_undegraded":"2024-09-21T23:26:07.515275+0000","last_fullsized":"2024-09-21T23:26:07.515275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:01:38.740017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095847300000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":56,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-21T23:26:10.350833+0000","last_change":"2024-09-21T23:25:30.263343+0000","last_active":"2024-09-21T23:26:10.350833+0000","last_peered":"2024-09-21T23:26:10.350833+0000","last_clean":"2024-09-21T23:26:10.350833+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:26:10.350833+0000","last_undegraded":"2024-09-21T23:26:10.350833+0000","last_fullsized":"2024-09-21T23:26:10.350833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:50:16.699815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044148899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":55,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-21T23:26:07.341932+0000","last_change":"2024-09-21T23:25:30.263501+0000","last_active":"2024-09-21T23:26:07.341932+0000","last_peered":"2024-09-21T23:26:07.341932+0000","last_clean":"2024-09-21T23:26:07.341932+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:26:07.341932+0000","last_undegraded":"2024-09-21T23:26:07.341932+0000","last_fullsized":"2024-09-21T23:26:07.341932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.155893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036082099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":55,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-21T23:26:07.515280+0000","last_change":"2024-09-21T23:25:30.186477+0000","last_active":"2024-09-21T23:26:07.515280+0000","last_peered":"2024-09-21T23:26:07.515280+0000","last_clean":"2024-09-21T23:26:07.515280+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:26:07.515280+0000","last_undegraded":"2024-09-21T23:26:07.515280+0000","last_fullsized":"2024-09-21T23:26:07.515280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:32:47.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044681799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-21T23:26:07.342048+0000","last_change":"2024-09-21T23:26:01.315848+0000","last_active":"2024-09-21T23:26:07.342048+0000","last_peered":"2024-09-21T23:26:07.342048+0000","last_clean":"2024-09-21T23:26:07.342048+0000","last_became_active":"2024-09-21T23:26:01.315355+0000","last_became_peered":"2024-09-21T23:26:01.315355+0000","last_unstale":"2024-09-21T23:26:07.342048+0000","last_undegraded":"2024-09-21T23:26:07.342048+0000","last_fullsized":"2024-09-21T23:26:07.342048+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:39:32.215042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":2361472,"data_compressed":37034,"data_compressed_allocated":147456,"data_compressed_original":2326528,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542158,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31608,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340232,"statfs":{"total":96636764160,"available":96604397568,"internally_reserved":0,"allocated":430080,"data_stored":169601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509455,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":434176,"data_stored":169620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":483328,"data_stored":765469,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509454,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":442368,"data_stored":175101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509455,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":434176,"data_stored":169620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":442368,"data_stored":175101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31608,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340232,"statfs":{"total":96636764160,"available":96604397568,"internally_reserved":0,"allocated":430080,"data_stored":169601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":471040,"data_stored":759969,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:26:15.739 INFO:tasks.thrashosds.thrasher:Setting 2.6 to [2, 4] 2024-09-21T23:26:15.739 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.6', '2', '4'] 2024-09-21T23:26:15.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.6 2 4 2024-09-21T23:26:15.823 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:15.823+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:15.908 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:15.908+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:15.914 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:15.914 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:15.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:16.011 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:16.011+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:16.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:26:16.082 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:16.083 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:16.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:16.135 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:16.134+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:16.211 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:16.210+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:16.233 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:16.233 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:16.267 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:16.268 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:16.297 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:16.297 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:16.300 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:16.300 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:16.309 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:16.312 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:16.310+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:16.332 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:16.333 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:16.364 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:set 2.6 pg_upmap mapping to [2,4] 2024-09-21T23:26:16.390 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-09-21T23:26:16.396 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:16.396 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:16.412 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:16.411+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:16.427 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:16.428 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:16.457 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:16.458 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:16.474 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:16.474 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:16.474 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:16.474 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:16.474 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:16.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:16.514 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:16.512+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:16.614 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:16.613+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:16.639 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:16.639 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:16.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:16.639 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:16.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.639 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-21T23:26:16.640 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:16.640 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 47.468772025, 2024-09-21T23:26:16.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003527526, 2024-09-21T23:26:16.640 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:16.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:16.640 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:16.640 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:16.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:16.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:16.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170743+0000", 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.08e-07 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:16.641 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170750+0000", 2024-09-21T23:26:16.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800000000000001e-06 2024-09-21T23:26:16.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:16.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170767+0000", 2024-09-21T23:26:16.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6923999999999999e-05 2024-09-21T23:26:16.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:16.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170788+0000", 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0446e-05 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170849+0000", 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1409999999999996e-05 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170881+0000", 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2075e-05 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:16.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174218+0000", 2024-09-21T23:26:16.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033371070000000002 2024-09-21T23:26:16.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:16.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174270+0000", 2024-09-21T23:26:16.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1375999999999997e-05 2024-09-21T23:26:16.644 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:16.644 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:16.644 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:16.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 46.452782530999997, 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011783570000000001, 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:16.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7999999999999996e-07 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186750+0000", 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7444000000000001e-05 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186834+0000", 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3974999999999995e-05 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186876+0000", 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2796000000000002e-05 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186999+0000", 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012303800000000001 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187036+0000", 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6897000000000003e-05 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187869+0000", 2024-09-21T23:26:16.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00083215199999999996 2024-09-21T23:26:16.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:16.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:16.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:16.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187910+0000", 2024-09-21T23:26:16.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1575000000000001e-05 2024-09-21T23:26:16.649 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:16.649 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:16.649 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:16.649 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:16.649 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:16.649 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:16.650 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:16.715 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:16.714+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:16.761 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:16.762 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:16.762 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:16.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:16.854 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:16.854+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:16.916 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:16.915+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:16.920 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:16.920 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:16.920 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:16.920 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:16.920 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:16.930 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:17.016 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:17.016+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:17.078 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:17.079 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:17.079 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:17.079 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:17.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.079 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:1 2.3 e19/18 2:c4c92e5a:::rbd_trash:head v 19'1, mlcod=0'0)", 2024-09-21T23:26:17.079 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:17.079 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 48.911224986000001, 2024-09-21T23:26:17.079 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039243489999999997, 2024-09-21T23:26:17.079 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:17.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2999999999999999e-07 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167667+0000", 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7169999999999997e-06 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167679+0000", 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.276e-05 2024-09-21T23:26:17.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167688+0000", 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8380000000000008e-06 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167739+0000", 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0964999999999997e-05 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167762+0000", 2024-09-21T23:26:17.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2221999999999998e-05 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171515+0000", 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037531119999999999 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171582+0000", 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7504999999999998e-05 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:17.083 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4224.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:13.799177+0000", 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.2797058350000001, 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00027751000000000002, 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4224", 2024-09-21T23:26:17.084 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.60:0/3116157232", 2024-09-21T23:26:17.085 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 2 2024-09-21T23:26:17.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.085 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:17.085 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.085 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:17.085 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799177+0000", 2024-09-21T23:26:17.085 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:17.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.085 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.085 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:17.085 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799177+0000", 2024-09-21T23:26:17.086 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:17.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.086 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.086 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:17.086 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799179+0000", 2024-09-21T23:26:17.086 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.956e-06 2024-09-21T23:26:17.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.086 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.086 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799196+0000", 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6646999999999998e-05 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799213+0000", 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7133e-05 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799225+0000", 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2044e-05 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799327+0000", 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000101983 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799454+0000", 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000127747 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:17.088 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:17.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:17.117 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:17.116+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:17.199 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:17.199 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:17.199 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:17.199 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:17.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:17.253 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:17.253+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:17.319 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:17.318+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:17.320 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:17.320 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:17.320 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:17.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:17.420 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:17.420+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:17.483 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:17.483 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:17.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:17.483 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:17.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.483 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:17.483 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:17.483 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 18.181156483999999, 2024-09-21T23:26:17.483 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00081488299999999997, 2024-09-21T23:26:17.483 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:17.483 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.484 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301851+0000", 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7899999999999998e-07 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302047+0000", 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019639200000000001 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302064+0000", 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7380999999999998e-05 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.485 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:17.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302080+0000", 2024-09-21T23:26:17.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5237e-05 2024-09-21T23:26:17.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:17.486 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302153+0000", 2024-09-21T23:26:17.486 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3922000000000001e-05 2024-09-21T23:26:17.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.486 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.486 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302162+0000", 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8170000000000005e-06 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302665+0000", 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050245499999999996 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:17.487 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 18.178373266000001, 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00068581899999999995, 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:17.488 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304634+0000", 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5499999999999998e-07 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304796+0000", 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016237700000000001 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:17.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304813+0000", 2024-09-21T23:26:17.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.732e-05 2024-09-21T23:26:17.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.490 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:17.490 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304825+0000", 2024-09-21T23:26:17.490 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1302e-05 2024-09-21T23:26:17.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304896+0000", 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1729000000000004e-05 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304905+0000", 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8359999999999998e-06 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.305319+0000", 2024-09-21T23:26:17.491 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00041360000000000002 2024-09-21T23:26:17.492 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:17.492 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:17.492 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:17.492 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:17.492 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:17.492 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:17.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:17.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:17.534+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:17.615 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:17.615 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:17.615 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:17.615 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:17.621 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:17.621+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:17.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:17.722 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:17.722+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:17.732 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:17.732 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:17.732 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:17.732 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:17.732 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:17.741 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:17.822 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:17.822+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:17.848 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:17.848 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:17.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:17.848 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.546872649000001, 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00108676, 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:17.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:17.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:17.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:17.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:17.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:17.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301959+0000", 2024-09-21T23:26:17.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3099999999999996e-07 2024-09-21T23:26:17.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302322+0000", 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00036268299999999998 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302345+0000", 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3278e-05 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302367+0000", 2024-09-21T23:26:17.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2118999999999999e-05 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302468+0000", 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100579 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302479+0000", 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1430999999999999e-05 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:17.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.303045+0000", 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000565739 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.544116411000001, 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010194869999999999, 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:17.853 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:17.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304715+0000", 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9599999999999998e-07 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305026+0000", 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031072100000000002 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305048+0000", 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1875999999999999e-05 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:17.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305066+0000", 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8644e-05 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305163+0000", 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6333000000000001e-05 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305172+0000", 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2189999999999993e-06 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305734+0000", 2024-09-21T23:26:17.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056179799999999996 2024-09-21T23:26:17.857 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:17.857 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:17.857 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:17.857 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:17.857 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:17.857 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:17.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:17.923 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:17.922+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:17.965 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:17.965 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:17.965 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:17.965 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:17.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:18.057 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:18.056+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:18.121 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:18.121 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:18.121 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:18.121 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:18.121 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:18.124 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:18.123+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:18.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:18.225 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:18.225+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:18.278 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:18.278 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:18.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.114372971999998, 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0080986310000000002, 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:18.279 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:18.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163709+0000", 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163727+0000", 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7552000000000001e-05 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163744+0000", 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7541999999999999e-05 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163757+0000", 2024-09-21T23:26:18.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2855e-05 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164069+0000", 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031179899999999997 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164084+0000", 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5265e-05 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167127+0000", 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030427029999999999 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167317+0000", 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000190006 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167465+0000", 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014824 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.168309+0000", 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084350899999999999 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171730+0000", 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034217620000000001 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171770+0000", 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0074999999999998e-05 2024-09-21T23:26:18.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171802+0000", 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1563999999999998e-05 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:1 2.3 e19/18 ondisk, result = 0)", 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.106430725000003, 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016281800000000001, 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:18.285 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:18.286 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:18.286 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:18.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:18.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:18.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:18.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:18.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:18.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:18.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6899999999999996e-07 2024-09-21T23:26:18.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:18.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171651+0000", 2024-09-21T23:26:18.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0309999999999998e-06 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171665+0000", 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.309e-05 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171672+0000", 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6070000000000002e-06 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171716+0000", 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4344000000000001e-05 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171727+0000", 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0488e-05 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171808+0000", 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1489000000000005e-05 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:18.289 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.290 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:18.290 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:18.290 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.103655283000002, 2024-09-21T23:26:18.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023803299999999999, 2024-09-21T23:26:18.290 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:18.290 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:18.290 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:18.290 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 2 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:18.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174422+0000", 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4300000000000001e-06 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174429+0000", 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1659999999999999e-06 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174443+0000", 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4358e-05 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174449+0000", 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2770000000000002e-06 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174540+0000", 2024-09-21T23:26:18.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0209000000000003e-05 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174659+0000", 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011959300000000001 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.103190283000004, 2024-09-21T23:26:18.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016795100000000001, 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 3 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:18.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:18.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:18.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174887+0000", 2024-09-21T23:26:18.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-09-21T23:26:18.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174892+0000", 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8180000000000003e-06 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174902+0000", 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0095000000000001e-05 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174911+0000", 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0799999999999995e-06 2024-09-21T23:26:18.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174974+0000", 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3027000000000005e-05 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.175054+0000", 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0011000000000004e-05 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 49.109684651999999, 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006147263, 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168394+0000", 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.956e-06 2024-09-21T23:26:18.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168404+0000", 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0463e-05 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168423+0000", 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8715999999999998e-05 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:18.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168432+0000", 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.516e-06 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168529+0000", 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7073999999999999e-05 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168534+0000", 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.831e-06 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170224+0000", 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016893590000000001 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170332+0000", 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108346 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170445+0000", 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112642 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:18.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.171210+0000", 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076543800000000001 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174469+0000", 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003259086 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174506+0000", 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6465999999999999e-05 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174539+0000", 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3370000000000001e-05 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:5 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 49.103742607000001, 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021298500000000001, 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:18.307 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:18.308 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:18.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:18.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:18.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:18.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:18.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:18.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174335+0000", 2024-09-21T23:26:18.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.111e-06 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174341+0000", 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6239999999999996e-06 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174356+0000", 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4548e-05 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174375+0000", 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8757e-05 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174457+0000", 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2702999999999997e-05 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:18.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174466+0000", 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.208e-06 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174547+0000", 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1033999999999997e-05 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:18.312 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 48.093970568000003, 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040438610000000002, 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184109+0000", 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9069999999999999e-06 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:18.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184124+0000", 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5780000000000001e-05 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184159+0000", 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4106999999999997e-05 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184178+0000", 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9908999999999999e-05 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184330+0000", 2024-09-21T23:26:18.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015152300000000001 2024-09-21T23:26:18.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:18.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184337+0000", 2024-09-21T23:26:18.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7920000000000004e-06 2024-09-21T23:26:18.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186239+0000", 2024-09-21T23:26:18.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019020129999999999 2024-09-21T23:26:18.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:18.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186307+0000", 2024-09-21T23:26:18.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8326999999999998e-05 2024-09-21T23:26:18.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:18.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186459+0000", 2024-09-21T23:26:18.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152099 2024-09-21T23:26:18.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:18.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187261+0000", 2024-09-21T23:26:18.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00080186199999999997 2024-09-21T23:26:18.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:18.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188085+0000", 2024-09-21T23:26:18.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00082417199999999997 2024-09-21T23:26:18.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:18.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188120+0000", 2024-09-21T23:26:18.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5194999999999998e-05 2024-09-21T23:26:18.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:18.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188150+0000", 2024-09-21T23:26:18.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9175000000000001e-05 2024-09-21T23:26:18.321 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.321 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:18.321 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.322 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:7 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:18.322 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:18.322 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 48.090099832, 2024-09-21T23:26:18.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000179505, 2024-09-21T23:26:18.322 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:18.322 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:18.322 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:18.323 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:18.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:18.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:18.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:18.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:18.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:18.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:18.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.16e-07 2024-09-21T23:26:18.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187985+0000", 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3039999999999999e-06 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188000+0000", 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5097e-05 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188007+0000", 2024-09-21T23:26:18.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1330000000000001e-06 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188074+0000", 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7514000000000004e-05 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188082+0000", 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3649999999999998e-06 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:18.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188156+0000", 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4176000000000003e-05 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4224.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.4826231100000005, 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022720800000000001, 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:18.327 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4224", 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/3116157232", 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:18.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795455+0000", 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0609999999999998e-06 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795473+0000", 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7762e-05 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795487+0000", 2024-09-21T23:26:18.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.412e-05 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795496+0000", 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8780000000000004e-06 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795578+0000", 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1700999999999996e-05 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795680+0000", 2024-09-21T23:26:18.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102686 2024-09-21T23:26:18.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.331 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:18.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:18.331 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:18.331 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:18.333 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:18.333 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:18.331+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:18.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T23:26:18.455 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:18.455+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:18.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:26:18.480 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:18.480 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:18.480 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:18.480 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:18.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:18.533 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:18.532+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:18.596 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:18.596 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:18.596 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:18.596 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:18.596 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:18.605 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:18.636 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:18.635+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:18.678 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:18.678 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:18.711 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:18.736 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:18.735+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:18.743 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:18.743 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:18.752 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:18.752 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:18.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:18.752 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-09-21T23:26:18.752 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:18.761 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:18.775 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:18.775 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:18.805 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:18.830 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:18.831 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:18.837 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:18.836+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:18.862 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:18.892 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:18.910 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:18.910 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:18.918 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:18.963 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:18.962+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:19.025 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:19.025 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:19.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:19.038 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:19.038+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:19.140 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:19.139+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:19.180 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:19.180 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:19.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:19.241 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:19.240+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:19.301 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:19.301 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:19.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:19.352 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:19.352+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:19.400 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-09-21T23:26:19.442 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:19.442+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:19.469 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:19.469 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:19.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:19.543 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:19.543+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:19.589 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:19.589 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:19.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:19.644 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:19.644+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:19.711 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:19.711 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:19.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:19.764 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:19.764+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:19.830 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:19.830 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:19.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:19.847 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:19.846+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:19.948 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:19.947+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:19.995 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:19.995 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:20.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:20.048 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:20.047+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:20.116 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:20.116 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:20.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:20.149 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:20.148+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:20.250 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:20.249+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:20.260 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:20.260 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:20.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:20.351 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:20.350+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:20.424 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:20.424 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:20.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:20.477 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:20.477+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:20.546 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:20.546 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:20.553 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:20.552+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:20.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:20.654 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:20.653+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:20.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:20.758 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:20.758+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:20.855 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:20.855+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:20.867 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:20.867 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:20.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:20.908 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:26:20.960 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:20.960+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:20.997 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:20.997 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:21.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:21.043 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:21.044 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:21.057 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:21.056+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:21.075 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:21.076 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:21.106 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:21.106 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:21.115 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:21.115 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:21.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:21.136 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:21.137 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:21.154 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:21.154 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:21.165 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:21.164+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:21.184 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:21.203 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:21.233 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:21.233 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:21.259 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:21.258+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:21.273 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 21.971224847999999, 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00108676, 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301959+0000", 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3099999999999996e-07 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302322+0000", 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00036268299999999998 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302345+0000", 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3278e-05 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302367+0000", 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2118999999999999e-05 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302468+0000", 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100579 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302479+0000", 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1430999999999999e-05 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.303045+0000", 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000565739 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 21.968468609999999, 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010194869999999999, 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304715+0000", 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9599999999999998e-07 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305026+0000", 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031072100000000002 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305048+0000", 2024-09-21T23:26:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1875999999999999e-05 2024-09-21T23:26:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305066+0000", 2024-09-21T23:26:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8644e-05 2024-09-21T23:26:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305163+0000", 2024-09-21T23:26:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6333000000000001e-05 2024-09-21T23:26:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305172+0000", 2024-09-21T23:26:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2189999999999993e-06 2024-09-21T23:26:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305734+0000", 2024-09-21T23:26:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056179799999999996 2024-09-21T23:26:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.283 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:21.284 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:21.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:21.359+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:21.401 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:26:21.401 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:26:21.401 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T23:26:21.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T23:26:21.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T23:26:21.431 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:21.431 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:21.431 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:21.432 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:21.440 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:21.490 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:21.490+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:21.547 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:21.547 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:21.547 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:21.547 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:21.547 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:21.556 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:21.600 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:21.599+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:21.663 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:21.661+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:21.664 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:21.664 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:21.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:21.664 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:21.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 53.500392109000003, 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0080986310000000002, 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:21.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163709+0000", 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-09-21T23:26:21.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163727+0000", 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7552000000000001e-05 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163744+0000", 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7541999999999999e-05 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163757+0000", 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2855e-05 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164069+0000", 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031179899999999997 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164084+0000", 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5265e-05 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167127+0000", 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030427029999999999 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167317+0000", 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000190006 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167465+0000", 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014824 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.168309+0000", 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084350899999999999 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171730+0000", 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034217620000000001 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171770+0000", 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0074999999999998e-05 2024-09-21T23:26:21.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171802+0000", 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1563999999999998e-05 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:1 2.3 e19/18 ondisk, result = 0)", 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 53.492449862000001, 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016281800000000001, 2024-09-21T23:26:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6899999999999996e-07 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171651+0000", 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0309999999999998e-06 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171665+0000", 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.309e-05 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171672+0000", 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6070000000000002e-06 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171716+0000", 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4344000000000001e-05 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171727+0000", 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0488e-05 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171808+0000", 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1489000000000005e-05 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 53.48967442, 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023803299999999999, 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:21.675 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 2 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:21.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174422+0000", 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4300000000000001e-06 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174429+0000", 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1659999999999999e-06 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:21.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174443+0000", 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4358e-05 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174449+0000", 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2770000000000002e-06 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174540+0000", 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0209000000000003e-05 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174659+0000", 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011959300000000001 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 53.489209420000002, 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016795100000000001, 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:21.679 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 3 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:21.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174887+0000", 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174892+0000", 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8180000000000003e-06 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174902+0000", 2024-09-21T23:26:21.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0095000000000001e-05 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174911+0000", 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0799999999999995e-06 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174974+0000", 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3027000000000005e-05 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.175054+0000", 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0011000000000004e-05 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 52.495703788999997, 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006147263, 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:21.683 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:21.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168394+0000", 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.956e-06 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168404+0000", 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0463e-05 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168423+0000", 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8715999999999998e-05 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168432+0000", 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.516e-06 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168529+0000", 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7073999999999999e-05 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168534+0000", 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.831e-06 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170224+0000", 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016893590000000001 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170332+0000", 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108346 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170445+0000", 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112642 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.171210+0000", 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076543800000000001 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174469+0000", 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003259086 2024-09-21T23:26:21.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174506+0000", 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6465999999999999e-05 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174539+0000", 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3370000000000001e-05 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:5 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:21.689 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 52.489761743999999, 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021298500000000001, 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:21.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174335+0000", 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.111e-06 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174341+0000", 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6239999999999996e-06 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:21.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174356+0000", 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4548e-05 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174375+0000", 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8757e-05 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174457+0000", 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2702999999999997e-05 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:21.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174466+0000", 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.208e-06 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174547+0000", 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1033999999999997e-05 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 51.479989705000001, 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040438610000000002, 2024-09-21T23:26:21.693 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184109+0000", 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9069999999999999e-06 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184124+0000", 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5780000000000001e-05 2024-09-21T23:26:21.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184159+0000", 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4106999999999997e-05 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184178+0000", 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9908999999999999e-05 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184330+0000", 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015152300000000001 2024-09-21T23:26:21.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184337+0000", 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7920000000000004e-06 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186239+0000", 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019020129999999999 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186307+0000", 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8326999999999998e-05 2024-09-21T23:26:21.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186459+0000", 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152099 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187261+0000", 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00080186199999999997 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188085+0000", 2024-09-21T23:26:21.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00082417199999999997 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188120+0000", 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5194999999999998e-05 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188150+0000", 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9175000000000001e-05 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:7 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 51.476118968999998, 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000179505, 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.16e-07 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187985+0000", 2024-09-21T23:26:21.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3039999999999999e-06 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188000+0000", 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5097e-05 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188007+0000", 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1330000000000001e-06 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188074+0000", 2024-09-21T23:26:21.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7514000000000004e-05 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188082+0000", 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3649999999999998e-06 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188156+0000", 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4176000000000003e-05 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4224.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 7.8686422470000004, 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022720800000000001, 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4224", 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/3116157232", 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:21.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795455+0000", 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0609999999999998e-06 2024-09-21T23:26:21.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795473+0000", 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7762e-05 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795487+0000", 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.412e-05 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795496+0000", 2024-09-21T23:26:21.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8780000000000004e-06 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795578+0000", 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1700999999999996e-05 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795680+0000", 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102686 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.707 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.708 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:21.708 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:21.708 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:21.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:21.722 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:26:21.722 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 44, 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "d51cc561-ddb8-41ef-b61c-7c6d698e4dd0", 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-09-21T23:25:09.056809+0000", 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-09-21T23:26:19.392622+0000", 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-09-21T23:25:20.126996+0000", 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-09-21T23:25:11.226810+0000", 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5804032, 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "nodeep-scrub", 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-09-21T23:26:21.723 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 5, 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 2, 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 8, 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "luminous", 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-09-21T23:26:21.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:19.818212+0000", 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:21.725 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 1, 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 1, 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 1, 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 1, 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 1, 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:21.726 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "17", 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:21.727 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:21.728 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:21.729 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 6.9000000953674316, 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 6.9000000953674316, 2024-09-21T23:26:21.730 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.28999999165534973, 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 2, 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "rbd", 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:26.968096+0000", 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 8193, 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 8, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 8, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 8, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 8, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 8, 2024-09-21T23:26:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "21", 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 2, 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 21, 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd": {} 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 1.75, 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 1.75, 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 1.1399999856948853, 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:26:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af", 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 35, 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-09-21T23:26:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-09-21T23:26:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6817/2339966024", 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6819/2339966024", 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6823/2339966024", 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6821/2339966024", 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "88ee06a3-9a6a-41e8-9327-a7a61304c03d", 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6825/2889649822", 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6827/2889649822", 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6831/2889649822", 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6829/2889649822", 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05", 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 42, 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:21.751 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6809/864917651", 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6811/864917651", 2024-09-21T23:26:21.752 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6815/864917651", 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6813/864917651", 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "562e8c30-0622-4cb3-b5d1-f15b42a444bc", 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:21.753 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 36, 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.754 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.756 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.757 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6801/1713632981", 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6803/1713632981", 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6807/1713632981", 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6805/1713632981", 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7030ff68-4556-4ad5-abc9-be758f1caa85", 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:21.758 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.759 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6808", 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6809", 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.760 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6810", 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6811", 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.761 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6814", 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6815", 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6812", 2024-09-21T23:26:21.762 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6813", 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6809/2660753894", 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6811/2660753894", 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6815/2660753894", 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6813/2660753894", 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:21.763 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7169a163-f067-4385-b734-f58a3ba9e89e", 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:21.764 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6824", 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6825", 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.765 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6826", 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6827", 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:21.766 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6830", 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6831", 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6828", 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6829", 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6825/2320687649", 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6827/2320687649", 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6831/2320687649", 2024-09-21T23:26:21.768 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6829/2320687649", 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e511295d-2177-4dcb-a74b-b16b92534df6", 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:21.769 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6800", 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6801", 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:21.770 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6802", 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6803", 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.771 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6806", 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6807", 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:21.772 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6804", 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6805", 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6801/3490482162", 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6803/3490482162", 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3490482162", 2024-09-21T23:26:21.773 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6805/3490482162", 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "544f8a91-ce82-4855-85eb-ea814eb3d058", 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:21.774 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6816", 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6817", 2024-09-21T23:26:21.775 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:21.776 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.776 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.776 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:21.776 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.777 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.777 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6818", 2024-09-21T23:26:21.777 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:21.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6819", 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6822", 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6823", 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6820", 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:21.779 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6821", 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6817/1315554966", 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6819/1315554966", 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6823/1315554966", 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6821/1315554966", 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.780 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.514872+0000", 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:21.781 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.912129+0000", 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.745154+0000", 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:21.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:16.758871+0000", 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:21.783 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.287462+0000", 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:18.031636+0000", 2024-09-21T23:26:21.784 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.055855+0000", 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:21.785 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.524609+0000", 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [ 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: 3, 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: 4 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.786 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "2.6", 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: 2, 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: 4 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [ 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "2.0", 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: "mappings": [ 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: "from": 1, 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: "to": 7 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.787 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [], 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": {}, 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-09-21T23:26:21.788 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:21.789 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:21.792 INFO:tasks.thrashosds.thrasher:j is {'epoch': 44, 'fsid': 'd51cc561-ddb8-41ef-b61c-7c6d698e4dd0', 'created': '2024-09-21T23:25:09.056809+0000', 'modified': '2024-09-21T23:26:19.392622+0000', 'last_up_change': '2024-09-21T23:25:20.126996+0000', 'last_in_change': '2024-09-21T23:25:11.226810+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T23:25:19.818212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T23:25:26.968096+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 2339966024}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 2339966024}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 2339966024}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 2339966024}]}, 'public_addr': '172.21.15.31:6817/2339966024', 'cluster_addr': '172.21.15.31:6819/2339966024', 'heartbeat_back_addr': '172.21.15.31:6823/2339966024', 'heartbeat_front_addr': '172.21.15.31:6821/2339966024', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '88ee06a3-9a6a-41e8-9327-a7a61304c03d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 2889649822}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2889649822}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2889649822}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2889649822}]}, 'public_addr': '172.21.15.31:6825/2889649822', 'cluster_addr': '172.21.15.31:6827/2889649822', 'heartbeat_back_addr': '172.21.15.31:6831/2889649822', 'heartbeat_front_addr': '172.21.15.31:6829/2889649822', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 42, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 864917651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 864917651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 864917651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 864917651}]}, 'public_addr': '172.21.15.31:6809/864917651', 'cluster_addr': '172.21.15.31:6811/864917651', 'heartbeat_back_addr': '172.21.15.31:6815/864917651', 'heartbeat_front_addr': '172.21.15.31:6813/864917651', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '562e8c30-0622-4cb3-b5d1-f15b42a444bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 36, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6800', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6801', 'nonce': 1713632981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1713632981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1713632981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1713632981}]}, 'public_addr': '172.21.15.31:6801/1713632981', 'cluster_addr': '172.21.15.31:6803/1713632981', 'heartbeat_back_addr': '172.21.15.31:6807/1713632981', 'heartbeat_front_addr': '172.21.15.31:6805/1713632981', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7030ff68-4556-4ad5-abc9-be758f1caa85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6809', 'nonce': 2660753894}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6811', 'nonce': 2660753894}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6815', 'nonce': 2660753894}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6813', 'nonce': 2660753894}]}, 'public_addr': '172.21.15.60:6809/2660753894', 'cluster_addr': '172.21.15.60:6811/2660753894', 'heartbeat_back_addr': '172.21.15.60:6815/2660753894', 'heartbeat_front_addr': '172.21.15.60:6813/2660753894', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7169a163-f067-4385-b734-f58a3ba9e89e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6824', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6825', 'nonce': 2320687649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6826', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6827', 'nonce': 2320687649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6830', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6831', 'nonce': 2320687649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6828', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6829', 'nonce': 2320687649}]}, 'public_addr': '172.21.15.60:6825/2320687649', 'cluster_addr': '172.21.15.60:6827/2320687649', 'heartbeat_back_addr': '172.21.15.60:6831/2320687649', 'heartbeat_front_addr': '172.21.15.60:6829/2320687649', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e511295d-2177-4dcb-a74b-b16b92534df6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6801', 'nonce': 3490482162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6803', 'nonce': 3490482162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6807', 'nonce': 3490482162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6805', 'nonce': 3490482162}]}, 'public_addr': '172.21.15.60:6801/3490482162', 'cluster_addr': '172.21.15.60:6803/3490482162', 'heartbeat_back_addr': '172.21.15.60:6807/3490482162', 'heartbeat_front_addr': '172.21.15.60:6805/3490482162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '544f8a91-ce82-4855-85eb-ea814eb3d058', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6816', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6817', 'nonce': 1315554966}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6818', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6819', 'nonce': 1315554966}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6822', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6823', 'nonce': 1315554966}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6820', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6821', 'nonce': 1315554966}]}, 'public_addr': '172.21.15.60:6817/1315554966', 'cluster_addr': '172.21.15.60:6819/1315554966', 'heartbeat_back_addr': '172.21.15.60:6823/1315554966', 'heartbeat_front_addr': '172.21.15.60:6821/1315554966', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.514872+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.912129+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.745154+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:16.758871+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.287462+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:18.031636+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.055855+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.524609+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [3, 4]}, {'pgid': '2.6', 'osds': [2, 4]}], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 1, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T23:26:21.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:26:21.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:21.790+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:21.891 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:21.890+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:21.897 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:21.897 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:21.897 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:21.897 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:21.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:21.992 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:21.991+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:22.054 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:22.054 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:22.054 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:22.054 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:22.054 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:22.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:22.104 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:22.103+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:22.131 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:26:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:26:22.147 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-09-21T23:26:21.784983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236780,"kb_used_data":3904,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737940,"statfs":{"total":773094113280,"available":772851650560,"internally_reserved":0,"allocated":3997696,"data_stored":2747516,"data_compressed":19550,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000933"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":63,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-21T23:26:17.380022+0000","last_change":"2024-09-21T23:25:30.180125+0000","last_active":"2024-09-21T23:26:17.380022+0000","last_peered":"2024-09-21T23:26:17.380022+0000","last_clean":"2024-09-21T23:26:17.380022+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:26:17.380022+0000","last_undegraded":"2024-09-21T23:26:17.380022+0000","last_fullsized":"2024-09-21T23:26:17.380022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:17.515296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033759699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-21T23:26:19.408413+0000","last_change":"2024-09-21T23:26:17.546748+0000","last_active":"2024-09-21T23:26:19.408413+0000","last_peered":"2024-09-21T23:26:19.408413+0000","last_clean":"2024-09-21T23:26:19.408413+0000","last_became_active":"2024-09-21T23:26:17.546387+0000","last_became_peered":"2024-09-21T23:26:17.546387+0000","last_unstale":"2024-09-21T23:26:19.408413+0000","last_undegraded":"2024-09-21T23:26:19.408413+0000","last_fullsized":"2024-09-21T23:26:19.408413+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:56:12.695613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053117699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-21T23:26:20.100954+0000","last_change":"2024-09-21T23:25:30.263565+0000","last_active":"2024-09-21T23:26:20.100954+0000","last_peered":"2024-09-21T23:26:20.100954+0000","last_clean":"2024-09-21T23:26:20.100954+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:26:20.100954+0000","last_undegraded":"2024-09-21T23:26:20.100954+0000","last_fullsized":"2024-09-21T23:26:20.100954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:46:37.545348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042085499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-21T23:26:20.100632+0000","last_change":"2024-09-21T23:25:30.189121+0000","last_active":"2024-09-21T23:26:20.100632+0000","last_peered":"2024-09-21T23:26:20.100632+0000","last_clean":"2024-09-21T23:26:20.100632+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:26:20.100632+0000","last_undegraded":"2024-09-21T23:26:20.100632+0000","last_fullsized":"2024-09-21T23:26:20.100632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:00:22.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054857199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":65,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-21T23:26:20.100903+0000","last_change":"2024-09-21T23:25:30.186984+0000","last_active":"2024-09-21T23:26:20.100903+0000","last_peered":"2024-09-21T23:26:20.100903+0000","last_clean":"2024-09-21T23:26:20.100903+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:26:20.100903+0000","last_undegraded":"2024-09-21T23:26:20.100903+0000","last_fullsized":"2024-09-21T23:26:20.100903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:01:38.740017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095847300000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-21T23:26:19.408354+0000","last_change":"2024-09-21T23:25:30.263343+0000","last_active":"2024-09-21T23:26:19.408354+0000","last_peered":"2024-09-21T23:26:19.408354+0000","last_clean":"2024-09-21T23:26:19.408354+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:26:19.408354+0000","last_undegraded":"2024-09-21T23:26:19.408354+0000","last_fullsized":"2024-09-21T23:26:19.408354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:50:16.699815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044148899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-21T23:26:20.100969+0000","last_change":"2024-09-21T23:25:30.263501+0000","last_active":"2024-09-21T23:26:20.100969+0000","last_peered":"2024-09-21T23:26:20.100969+0000","last_clean":"2024-09-21T23:26:20.100969+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:26:20.100969+0000","last_undegraded":"2024-09-21T23:26:20.100969+0000","last_fullsized":"2024-09-21T23:26:20.100969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.155893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036082099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-21T23:26:20.100920+0000","last_change":"2024-09-21T23:25:30.186477+0000","last_active":"2024-09-21T23:26:20.100920+0000","last_peered":"2024-09-21T23:26:20.100920+0000","last_clean":"2024-09-21T23:26:20.100920+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:26:20.100920+0000","last_undegraded":"2024-09-21T23:26:20.100920+0000","last_fullsized":"2024-09-21T23:26:20.100920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:32:47.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044681799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":186,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-21T23:26:17.544391+0000","last_change":"2024-09-21T23:26:01.315848+0000","last_active":"2024-09-21T23:26:17.544391+0000","last_peered":"2024-09-21T23:26:17.544391+0000","last_clean":"2024-09-21T23:26:17.544391+0000","last_became_active":"2024-09-21T23:26:01.315355+0000","last_became_peered":"2024-09-21T23:26:01.315355+0000","last_unstale":"2024-09-21T23:26:17.544391+0000","last_undegraded":"2024-09-21T23:26:17.544391+0000","last_fullsized":"2024-09-21T23:26:17.544391+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:39:32.215042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":2361472,"data_compressed":37034,"data_compressed_allocated":147456,"data_compressed_original":2326528,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542160,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":491520,"data_stored":197225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509457,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":197244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542159,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":532480,"data_stored":787593,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":491520,"data_stored":197225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27512,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":495616,"data_stored":197244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509456,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27512,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":491520,"data_stored":197225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":479232,"data_stored":191696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":520192,"data_stored":782064,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:26:22.148 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [0, 1] 2024-09-21T23:26:22.148 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '0', '1'] 2024-09-21T23:26:22.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 1.0 0 1 2024-09-21T23:26:22.194 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:22.193+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:22.210 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:22.210 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:22.210 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:22.210 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-09-21T23:26:22.210 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:22.219 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:22.295 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:22.295+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:22.326 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:22.326 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:22.326 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:22.326 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:22.334 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:22.396 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:22.395+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:22.415 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-09-21T23:26:22.442 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:22.442 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:22.443 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:22.443 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:22.443 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:22.451 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:22.497 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:22.496+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:22.597 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:22.596+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:22.600 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:22.600 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:22.600 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:22.600 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.431920443999999, 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090066050000000009, 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 37 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.601 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:22.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7099999999999998e-07 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167009+0000", 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.92e-06 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167013+0000", 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167017+0000", 2024-09-21T23:26:22.603 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7280000000000001e-06 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170778+0000", 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037612399999999999 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170831+0000", 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2694000000000001e-05 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170970+0000", 2024-09-21T23:26:22.604 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000138966 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.171555+0000", 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058538399999999999 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175872+0000", 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043172389999999996 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.605 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175969+0000", 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6844999999999999e-05 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.176012+0000", 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3356999999999998e-05 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.606 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.405815607999997, 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069023069999999999, 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0999999999999999e-07 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.608 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193115+0000", 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5700000000000001e-06 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193129+0000", 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4124000000000001e-05 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193133+0000", 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0550000000000001e-06 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.609 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194144+0000", 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001010735 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194231+0000", 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7665999999999997e-05 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194357+0000", 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125353 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194844+0000", 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000487305 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199964+0000", 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051204859999999996 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199992+0000", 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7902000000000002e-05 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.200013+0000", 2024-09-21T23:26:22.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0601000000000001e-05 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.384166534000002, 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0067494620000000003, 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.612 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 73 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.613 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4900000000000001e-07 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214818+0000", 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7754000000000003e-05 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214842+0000", 2024-09-21T23:26:22.614 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4552000000000001e-05 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214846+0000", 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9129999999999996e-06 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215813+0000", 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00096695899999999998 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.615 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215904+0000", 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0328000000000007e-05 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216020+0000", 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115838 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217018+0000", 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099836200000000003 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.616 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221461+0000", 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044435969999999997 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221487+0000", 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5806e-05 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221509+0000", 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1903999999999999e-05 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.617 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.384071831, 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068139230000000004, 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.618 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 74 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214855+0000", 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-07 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214859+0000", 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1389999999999997e-06 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214871+0000", 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1698999999999999e-05 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.620 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214887+0000", 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6061000000000001e-05 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216359+0000", 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014720779999999999 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216439+0000", 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0191999999999994e-05 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216579+0000", 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014029099999999999 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217129+0000", 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054932099999999997 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221559+0000", 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044299710000000004 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221627+0000", 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8631000000000001e-05 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221668+0000", 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0988000000000001e-05 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.623 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.367395645000002, 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0077112509999999997, 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 86 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:22.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5199999999999998e-07 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231533+0000", 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3879999999999998e-06 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231537+0000", 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0010000000000003e-06 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231540+0000", 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0489999999999999e-06 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233890+0000", 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023500249999999999 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233937+0000", 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6400999999999998e-05 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234015+0000", 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8343000000000005e-05 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235675+0000", 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001660124 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239138+0000", 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034624109999999999 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239200+0000", 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2698000000000002e-05 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239242+0000", 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1458999999999997e-05 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.629 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.367379376000002, 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010200097999999999, 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2700000000000001e-07 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231588+0000", 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0985000000000001e-05 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231593+0000", 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1059999999999999e-06 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231606+0000", 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2343e-05 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.632 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234406+0000", 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027999850000000001 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234489+0000", 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3051000000000002e-05 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234631+0000", 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285799999999999 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.633 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235679+0000", 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001047517 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241644+0000", 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059652129999999996 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241704+0000", 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9732999999999999e-05 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.634 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241747+0000", 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-05 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.367313537000001, 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010246534, 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.635 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 88 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.636 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0199999999999998e-07 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231615+0000", 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3439999999999999e-06 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.637 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231619+0000", 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0219999999999998e-06 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231622+0000", 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6989999999999999e-06 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235337+0000", 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037150030000000001 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.638 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235385+0000", 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7861999999999997e-05 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235458+0000", 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2596999999999996e-05 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235978+0000", 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052013600000000004 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.639 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241799+0000", 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058215289999999998 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241836+0000", 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6282999999999998e-05 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241859+0000", 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3757e-05 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.640 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.209922354, 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072206969999999999, 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 112 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.641 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:22.642 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.72e-07 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389069+0000", 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4875000000000005e-05 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389079+0000", 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0135e-05 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389085+0000", 2024-09-21T23:26:22.643 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394006+0000", 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049207920000000002 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394052+0000", 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5810999999999999e-05 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394139+0000", 2024-09-21T23:26:22.644 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7130000000000001e-05 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395407+0000", 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012681260000000001 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396139+0000", 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000731703 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396194+0000", 2024-09-21T23:26:22.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5309999999999997e-05 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396225+0000", 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0589000000000001e-05 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.209815397, 2024-09-21T23:26:22.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086923490000000003, 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 113 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389116+0000", 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.245e-06 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389126+0000", 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0295000000000001e-05 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389132+0000", 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0889999999999996e-06 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394548+0000", 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005415876 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394642+0000", 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4531999999999995e-05 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394714+0000", 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1753000000000002e-05 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396233+0000", 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015185299999999999 2024-09-21T23:26:22.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397749+0000", 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015157770000000001 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397779+0000", 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0561000000000001e-05 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397803+0000", 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4116000000000001e-05 2024-09-21T23:26:22.651 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.209783328, 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.01139824, 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.652 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 114 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.01e-07 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389146+0000", 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5239999999999999e-06 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389150+0000", 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3989999999999997e-06 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389153+0000", 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.038e-06 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395026+0000", 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058731720000000003 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395071+0000", 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4388999999999998e-05 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395141+0000", 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0053000000000002e-05 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397196+0000", 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020555899999999999 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400494+0000", 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032972679999999999 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400520+0000", 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6423000000000001e-05 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400541+0000", 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0982999999999999e-05 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.657 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.209764649, 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011537824, 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 115 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:22.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4900000000000001e-07 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389197+0000", 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5204999999999999e-05 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389203+0000", 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0979999999999999e-06 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389207+0000", 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6600000000000001e-06 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395443+0000", 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062358099999999996 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395525+0000", 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1855999999999997e-05 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395669+0000", 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014472399999999999 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397204+0000", 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001534817 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400599+0000", 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033944499999999998 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400655+0000", 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6647000000000002e-05 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400699+0000", 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4208000000000001e-05 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.209713196999999, 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011571580999999999, 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.663 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 116 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3099999999999999e-07 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389216+0000", 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4260000000000002e-06 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389223+0000", 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-09-21T23:26:22.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389225+0000", 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7760000000000002e-06 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396256+0000", 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070309020000000003 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396348+0000", 2024-09-21T23:26:22.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1780000000000006e-05 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396481+0000", 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000133101 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397208+0000", 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072716199999999995 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400737+0000", 2024-09-21T23:26:22.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035290220000000001 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400763+0000", 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5219999999999999e-05 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400785+0000", 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2045e-05 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.209696514000001, 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011687858000000001, 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 117 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3699999999999999e-07 2024-09-21T23:26:22.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389232+0000", 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1789999999999998e-06 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389238+0000", 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209999999999999e-06 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389241+0000", 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.557e-06 2024-09-21T23:26:22.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396792+0000", 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075506899999999997 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396838+0000", 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5639000000000001e-05 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396958+0000", 2024-09-21T23:26:22.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119892 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397458+0000", 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049987600000000001 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400824+0000", 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033662560000000002 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400880+0000", 2024-09-21T23:26:22.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5779999999999998e-05 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400918+0000", 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8031000000000002e-05 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.19767804, 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006886924, 2024-09-21T23:26:22.674 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 120 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401249+0000", 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.13e-07 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401286+0000", 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6773999999999997e-05 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401295+0000", 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9039999999999998e-06 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401299+0000", 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3250000000000001e-06 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402183+0000", 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000883758 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402264+0000", 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1080000000000003e-05 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402401+0000", 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013719700000000001 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403430+0000", 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010290119999999999 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408045+0000", 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046146659999999999 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408095+0000", 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0898e-05 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408135+0000", 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9697000000000003e-05 2024-09-21T23:26:22.679 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.197619566, 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069210920000000002, 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.680 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 121 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3200000000000001e-07 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401341+0000", 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4236999999999998e-05 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401346+0000", 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.04e-06 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401350+0000", 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6399999999999999e-06 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402733+0000", 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013825790000000001 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402816+0000", 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3461000000000006e-05 2024-09-21T23:26:22.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402941+0000", 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012518300000000001 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403533+0000", 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000592071 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408174+0000", 2024-09-21T23:26:22.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046404209999999996 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408204+0000", 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0147000000000001e-05 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408228+0000", 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3981e-05 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 60.761476668999997, 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064924400000000004, 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 139 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0800000000000001e-07 2024-09-21T23:26:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837452+0000", 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7209999999999999e-06 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837455+0000", 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2279999999999998e-06 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837458+0000", 2024-09-21T23:26:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8200000000000001e-06 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842187+0000", 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047298460000000002 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842233+0000", 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6056000000000001e-05 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842304+0000", 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0862e-05 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843291+0000", 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00098632100000000007 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843894+0000", 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060369199999999997 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843922+0000", 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7429000000000001e-05 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843942+0000", 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0256999999999999e-05 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:22.691 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 60.761464726, 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066155550000000004, 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 140 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:22.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1e-07 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837517+0000", 2024-09-21T23:26:22.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5034000000000002e-05 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837522+0000", 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8520000000000003e-06 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837525+0000", 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4139999999999998e-06 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842562+0000", 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050366279999999996 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842615+0000", 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3675999999999999e-05 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842715+0000", 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9990999999999999e-05 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843873+0000", 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011573 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844026+0000", 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015368900000000001 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844058+0000", 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1449999999999999e-05 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844077+0000", 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9310999999999999e-05 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 60.761392532999999, 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068324850000000001, 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.697 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 141 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.22e-07 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837539+0000", 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.023e-06 2024-09-21T23:26:22.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837543+0000", 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6490000000000001e-06 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837546+0000", 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.382e-06 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843479+0000", 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059328189999999998 2024-09-21T23:26:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843519+0000", 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0071000000000003e-05 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843600+0000", 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1376000000000002e-05 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844137+0000", 2024-09-21T23:26:22.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053640299999999997 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844319+0000", 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000181893 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844346+0000", 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.781e-05 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844366+0000", 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9737000000000001e-05 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 60.146058838000002, 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006594145, 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 158 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452868+0000", 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9500000000000003e-07 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452938+0000", 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9598999999999996e-05 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452950+0000", 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2971e-05 2024-09-21T23:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452960+0000", 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.448e-06 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456217+0000", 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003257378 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.706 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456557+0000", 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033989399999999998 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456703+0000", 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000145599 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459339+0000", 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026365529999999998 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459386+0000", 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6640000000000001e-05 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459428+0000", 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2148999999999999e-05 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459462+0000", 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3419e-05 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.708 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 60.145954463999999, 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0065672760000000004, 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 159 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.709 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:22.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5400000000000001e-07 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452977+0000", 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.301e-06 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452984+0000", 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1840000000000002e-06 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:22.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452989+0000", 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5219999999999998e-06 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457324+0000", 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043348780000000003 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457416+0000", 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1824999999999996e-05 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:22.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457574+0000", 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015810999999999999 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459376+0000", 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001801649 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459467+0000", 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1367999999999997e-05 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:22.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459507+0000", 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9576000000000002e-05 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459539+0000", 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2308999999999998e-05 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:22.714 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:22.716 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:22.715+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:22.716 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:22.817 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:22.816+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:22.827 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:22.827 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:22.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:22.922 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:22.922+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:22.996 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:22.996 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:22.996 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:22.996 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:22.996 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:23.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:23.046 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:23.046+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:23.120 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:23.119+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:23.160 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-09-21T23:26:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 62.022134792000003, 2024-09-21T23:26:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036127939999999999, 2024-09-21T23:26:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3500000000000002e-07 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136608+0000", 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.182e-06 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136618+0000", 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9199999999999999e-06 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136628+0000", 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4390000000000006e-06 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136692+0000", 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3979000000000005e-05 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136716+0000", 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4391000000000001e-05 2024-09-21T23:26:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140171+0000", 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034554910000000002 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140214+0000", 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2057000000000001e-05 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'18)", 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.987616629000001, 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043957730000000004, 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171120+0000", 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6700000000000003e-07 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171125+0000", 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.451e-06 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171138+0000", 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2541000000000001e-05 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171143+0000", 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4659999999999998e-06 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171187+0000", 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4014000000000002e-05 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171200+0000", 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3108999999999999e-05 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175479+0000", 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042784570000000003 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175515+0000", 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5967999999999999e-05 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:58 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.964864333000001, 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003032851, 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193872+0000", 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9400000000000004e-07 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193952+0000", 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9791999999999998e-05 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193962+0000", 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0034000000000001e-05 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193973+0000", 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0733e-05 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194015+0000", 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2911999999999998e-05 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194031+0000", 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5082999999999999e-05 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196869+0000", 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002838808 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196904+0000", 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4694999999999999e-05 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:59 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.964278667000002, 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053675709999999998, 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7100000000000002e-07 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194462+0000", 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.668e-06 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194469+0000", 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1670000000000002e-06 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194475+0000", 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-06 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194513+0000", 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7391999999999997e-05 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194531+0000", 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7870999999999999e-05 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199791+0000", 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052604119999999999 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199824+0000", 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3340000000000003e-05 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:60 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'38, mlcod=16'36)", 2024-09-21T23:26:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.958197972000001, 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003614562, 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200539+0000", 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5799999999999999e-06 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200551+0000", 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1816e-05 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200578+0000", 2024-09-21T23:26:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6772999999999999e-05 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200590+0000", 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2303000000000001e-05 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200715+0000", 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012523499999999999 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200731+0000", 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5690000000000001e-05 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204121+0000", 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003390363 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204152+0000", 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0802000000000001e-05 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:70 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'39)", 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.948142316999999, 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033896579999999998, 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210594+0000", 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1099999999999995e-07 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210600+0000", 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4389999999999997e-06 2024-09-21T23:26:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210613+0000", 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.243e-05 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210618+0000", 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0939999999999998e-06 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210843+0000", 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022493200000000001 2024-09-21T23:26:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210858+0000", 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5109000000000001e-05 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213938+0000", 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030796500000000002 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213983+0000", 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5293000000000002e-05 2024-09-21T23:26:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:72 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'46, mlcod=16'43)", 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.943197541000004, 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029102279999999999, 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215539+0000", 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4199999999999995e-07 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215589+0000", 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0488999999999997e-05 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215598+0000", 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2830000000000007e-06 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215610+0000", 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1134999999999999e-05 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215657+0000", 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7658999999999999e-05 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215679+0000", 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1526999999999999e-05 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218410+0000", 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027316939999999998 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218448+0000", 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7799000000000002e-05 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:73 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'47, mlcod=16'43)", 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.942588194000002, 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051882020000000003, 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216148+0000", 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7100000000000001e-07 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216218+0000", 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0313000000000003e-05 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216227+0000", 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9959999999999999e-06 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216232+0000", 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5340000000000001e-06 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216267+0000", 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4739000000000001e-05 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216281+0000", 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4345000000000001e-05 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221290+0000", 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050088989999999998 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221336+0000", 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5704999999999997e-05 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:74 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'48, mlcod=16'43)", 2024-09-21T23:26:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.942062251000003, 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046922939999999996, 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216674+0000", 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9900000000000001e-07 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216679+0000", 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.502e-06 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216687+0000", 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1790000000000008e-06 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216691+0000", 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7230000000000002e-06 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216737+0000", 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6087000000000001e-05 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216758+0000", 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0434000000000001e-05 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221338+0000", 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045804569999999996 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221366+0000", 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7413e-05 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:86 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'56, mlcod=16'50)", 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.924640316999998, 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049055269999999998, 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234096+0000", 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234099+0000", 2024-09-21T23:26:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3890000000000001e-06 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234107+0000", 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2349999999999999e-06 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234112+0000", 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0629999999999999e-06 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234143+0000", 2024-09-21T23:26:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1131999999999999e-05 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234155+0000", 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2140000000000001e-05 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.238972+0000", 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048167770000000004 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.239001+0000", 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9164000000000001e-05 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:87 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'57, mlcod=16'52)", 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.923960942000001, 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067179910000000004, 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0299999999999998e-07 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234813+0000", 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7531999999999999e-05 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234826+0000", 2024-09-21T23:26:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2617e-05 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234831+0000", 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1229999999999999e-06 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234869+0000", 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8513999999999998e-05 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234881+0000", 2024-09-21T23:26:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2148e-05 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241465+0000", 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065837329999999996 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241493+0000", 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7521e-05 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:88 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.923163094000003, 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059344530000000001, 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235573+0000", 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0100000000000004e-07 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235578+0000", 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1590000000000001e-06 2024-09-21T23:26:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235590+0000", 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1229e-05 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235595+0000", 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0889999999999999e-06 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235943+0000", 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000348227 2024-09-21T23:26:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235957+0000", 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4419999999999999e-05 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241494+0000", 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055371070000000003 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241507+0000", 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2520999999999999e-05 2024-09-21T23:26:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:113 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'75, mlcod=16'71)", 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.763931816000003, 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028543489999999999, 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8199999999999999e-07 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394809+0000", 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394821+0000", 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2024999999999999e-05 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394827+0000", 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0020000000000003e-06 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394862+0000", 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4811000000000003e-05 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394873+0000", 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.058e-05 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397631+0000", 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027581060000000002 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397658+0000", 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7073999999999999e-05 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'71)", 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.763520403000001, 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005160085, 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395216+0000", 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4999999999999998e-07 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395221+0000", 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2290000000000002e-06 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395229+0000", 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2369999999999992e-06 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395233+0000", 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1840000000000001e-06 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395260+0000", 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.728e-05 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395272+0000", 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1270000000000001e-05 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400341+0000", 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050691809999999999 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400375+0000", 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4254000000000001e-05 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'72)", 2024-09-21T23:26:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.762971882000002, 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046274580000000001, 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395829+0000", 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4618999999999999e-05 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395836+0000", 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9090000000000003e-06 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395839+0000", 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.371e-06 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395868+0000", 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9328000000000001e-05 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395880+0000", 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1501000000000001e-05 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400376+0000", 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044966340000000002 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400391+0000", 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4676e-05 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:116 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'78, mlcod=16'73)", 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.762167335000001, 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038400970000000002, 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396569+0000", 2024-09-21T23:26:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1100000000000001e-07 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396573+0000", 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5569999999999997e-06 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396579+0000", 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1120000000000001e-06 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396587+0000", 2024-09-21T23:26:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396630+0000", 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2991000000000002e-05 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396642+0000", 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1481000000000001e-05 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400392+0000", 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037507830000000002 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400408+0000", 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5863e-05 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:117 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'79, mlcod=16'73)", 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.761664992999997, 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033512120000000001, 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7e-07 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397075+0000", 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7809999999999999e-06 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397082+0000", 2024-09-21T23:26:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397085+0000", 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7139999999999999e-06 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397130+0000", 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.498e-05 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397142+0000", 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1838000000000001e-05 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400409+0000", 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032666349999999999 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400422+0000", 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2866e-05 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.756841319999999, 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034761240000000001, 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401895+0000", 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7400000000000003e-07 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401963+0000", 2024-09-21T23:26:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8701000000000005e-05 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401972+0000", 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9530000000000005e-06 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401977+0000", 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1579999999999998e-06 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402009+0000", 2024-09-21T23:26:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2512000000000003e-05 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402022+0000", 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3220999999999999e-05 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405337+0000", 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033150300000000001 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405370+0000", 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2975000000000002e-05 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.756235250000003, 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054168369999999999, 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402501+0000", 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2699999999999999e-07 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402589+0000", 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899000000000007e-05 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402599+0000", 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0312000000000001e-05 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402605+0000", 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9689999999999999e-06 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402648+0000", 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3210000000000001e-05 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402664+0000", 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5347999999999999e-05 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407887+0000", 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052237919999999997 2024-09-21T23:26:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407917+0000", 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9779999999999999e-05 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.755705869000003, 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049049180000000003, 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4099999999999999e-07 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403098+0000", 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7464000000000003e-05 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403106+0000", 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6629999999999999e-06 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403111+0000", 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403137+0000", 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6349000000000001e-05 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403148+0000", 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.077e-05 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407920+0000", 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047716590000000001 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407935+0000", 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4783999999999999e-05 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:23.247 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:23.248 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:26:23.249 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:23.248+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:23.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:23.349 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:23.349+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:23.414 INFO:teuthology.orchestra.run.smithi031.stderr:set 1.0 pg_upmap mapping to [0,1] 2024-09-21T23:26:23.421 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:23.421 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:23.421 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:23.421 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:23.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:23.450 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:23.449+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:23.477 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:23.477 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:23.498 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:23.498 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:23.521 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:23.551 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:23.550+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:23.555 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:23.555 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:23.588 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:23.588 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:23.622 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:23.650 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:23.650 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:23.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:23.651+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:23.657 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:23.657 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:23.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:23.675 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:23.675 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:23.752 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:23.751+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:23.784 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:23.784 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:23.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:23.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:23.852+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:23.907 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:23.907 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:23.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:23.954 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:23.953+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:24.055 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:24.054+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:24.064 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:24.064 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:24.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:24.157 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:24.157+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:24.222 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:24.222 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:24.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:24.272 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:24.272+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:24.357 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:24.356+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:24.382 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:24.382 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:24.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:24.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T23:26:24.479 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:24.479+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:24.506 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:24.506 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:24.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:24.558 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:24.557+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:24.659 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:24.658+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:24.667 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:24.667 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:24.677 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:24.760 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:24.759+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:24.797 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:24.797 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:24.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:24.848 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-09-21T23:26:24.861 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:24.861+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:24.916 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:24.916 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:24.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:24.968 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:24.967+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:25.031 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:25.031 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:25.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:25.081 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:25.080+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:25.164 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:25.164+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:25.187 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:25.187 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:25.196 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:25.265 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:25.264+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:25.302 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:25.302 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:25.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:25.365 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:25.365+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:25.459 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:25.466 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:25.465+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:25.468 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:25.567 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:25.566+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:25.616 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:25.616 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:25.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:25.668 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:25.668+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:25.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:26:25.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:25.769+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:25.772 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:25.772 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:25.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:25.841 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:25.841 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:25.870 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:25.870+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:25.873 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:25.888 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:25.888 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:25.896 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:25.903 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:25.903 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:25.934 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:25.934 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:25.965 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:25.966 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:25.971 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:25.970+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:25.996 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:25.996 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:26.006 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:26.006 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:26.015 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:26.027 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:26.027 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:26.059 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:26.059 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:26.071 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:26.070+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:26.123 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:26.123 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:26.124 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:26.124 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:26.124 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:26.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:26.173 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:26.172+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:26.273 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:26.273+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:26.281 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:26.281 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:26.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:26.281 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:26.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.281 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 65.112699816000003, 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090066050000000009, 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 37 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7099999999999998e-07 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167009+0000", 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.92e-06 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167013+0000", 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167017+0000", 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7280000000000001e-06 2024-09-21T23:26:26.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170778+0000", 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037612399999999999 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170831+0000", 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2694000000000001e-05 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170970+0000", 2024-09-21T23:26:26.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000138966 2024-09-21T23:26:26.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.171555+0000", 2024-09-21T23:26:26.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058538399999999999 2024-09-21T23:26:26.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175872+0000", 2024-09-21T23:26:26.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043172389999999996 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175969+0000", 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6844999999999999e-05 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.176012+0000", 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3356999999999998e-05 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 65.086594980000001, 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069023069999999999, 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0999999999999999e-07 2024-09-21T23:26:26.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193115+0000", 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5700000000000001e-06 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193129+0000", 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4124000000000001e-05 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193133+0000", 2024-09-21T23:26:26.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0550000000000001e-06 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194144+0000", 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001010735 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194231+0000", 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7665999999999997e-05 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194357+0000", 2024-09-21T23:26:26.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125353 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194844+0000", 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000487305 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199964+0000", 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051204859999999996 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199992+0000", 2024-09-21T23:26:26.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7902000000000002e-05 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.200013+0000", 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0601000000000001e-05 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 65.064945906000005, 2024-09-21T23:26:26.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0067494620000000003, 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 73 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4900000000000001e-07 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214818+0000", 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7754000000000003e-05 2024-09-21T23:26:26.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214842+0000", 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4552000000000001e-05 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214846+0000", 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9129999999999996e-06 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215813+0000", 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00096695899999999998 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215904+0000", 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0328000000000007e-05 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216020+0000", 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115838 2024-09-21T23:26:26.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217018+0000", 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099836200000000003 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221461+0000", 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044435969999999997 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221487+0000", 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5806e-05 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221509+0000", 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1903999999999999e-05 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 65.064851203000003, 2024-09-21T23:26:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068139230000000004, 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 74 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214855+0000", 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-07 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214859+0000", 2024-09-21T23:26:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1389999999999997e-06 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214871+0000", 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1698999999999999e-05 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214887+0000", 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6061000000000001e-05 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216359+0000", 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014720779999999999 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216439+0000", 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0191999999999994e-05 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216579+0000", 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014029099999999999 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217129+0000", 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054932099999999997 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221559+0000", 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044299710000000004 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221627+0000", 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8631000000000001e-05 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221668+0000", 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0988000000000001e-05 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 65.048175017000005, 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0077112509999999997, 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 86 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:26.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5199999999999998e-07 2024-09-21T23:26:26.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231533+0000", 2024-09-21T23:26:26.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3879999999999998e-06 2024-09-21T23:26:26.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231537+0000", 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0010000000000003e-06 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231540+0000", 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0489999999999999e-06 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233890+0000", 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023500249999999999 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233937+0000", 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6400999999999998e-05 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234015+0000", 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8343000000000005e-05 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235675+0000", 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001660124 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239138+0000", 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034624109999999999 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239200+0000", 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2698000000000002e-05 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239242+0000", 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1458999999999997e-05 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.310 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 65.048158748000006, 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010200097999999999, 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.311 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2700000000000001e-07 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231588+0000", 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0985000000000001e-05 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231593+0000", 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1059999999999999e-06 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231606+0000", 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2343e-05 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234406+0000", 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027999850000000001 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234489+0000", 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3051000000000002e-05 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234631+0000", 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285799999999999 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235679+0000", 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001047517 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241644+0000", 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059652129999999996 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241704+0000", 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9732999999999999e-05 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241747+0000", 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-05 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 65.048092909000005, 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010246534, 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 88 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:26.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:26.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0199999999999998e-07 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231615+0000", 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3439999999999999e-06 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231619+0000", 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0219999999999998e-06 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231622+0000", 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6989999999999999e-06 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235337+0000", 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037150030000000001 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235385+0000", 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7861999999999997e-05 2024-09-21T23:26:26.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235458+0000", 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2596999999999996e-05 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235978+0000", 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052013600000000004 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241799+0000", 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058215289999999998 2024-09-21T23:26:26.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241836+0000", 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6282999999999998e-05 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241859+0000", 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3757e-05 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.322 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.890701726000003, 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072206969999999999, 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 112 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.72e-07 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389069+0000", 2024-09-21T23:26:26.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4875000000000005e-05 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389079+0000", 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0135e-05 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389085+0000", 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394006+0000", 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049207920000000002 2024-09-21T23:26:26.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394052+0000", 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5810999999999999e-05 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394139+0000", 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7130000000000001e-05 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395407+0000", 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012681260000000001 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396139+0000", 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000731703 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396194+0000", 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5309999999999997e-05 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396225+0000", 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0589000000000001e-05 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.327 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.328 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.328 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:26.328 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.890594769000003, 2024-09-21T23:26:26.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086923490000000003, 2024-09-21T23:26:26.328 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.328 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.328 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.328 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 113 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389116+0000", 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.245e-06 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389126+0000", 2024-09-21T23:26:26.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0295000000000001e-05 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389132+0000", 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0889999999999996e-06 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394548+0000", 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005415876 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394642+0000", 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4531999999999995e-05 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394714+0000", 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1753000000000002e-05 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396233+0000", 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015185299999999999 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397749+0000", 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015157770000000001 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397779+0000", 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0561000000000001e-05 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397803+0000", 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4116000000000001e-05 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.333 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.890562700000004, 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.01139824, 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 114 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.01e-07 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389146+0000", 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5239999999999999e-06 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389150+0000", 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3989999999999997e-06 2024-09-21T23:26:26.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389153+0000", 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.038e-06 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395026+0000", 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058731720000000003 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395071+0000", 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4388999999999998e-05 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395141+0000", 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0053000000000002e-05 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397196+0000", 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020555899999999999 2024-09-21T23:26:26.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400494+0000", 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032972679999999999 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400520+0000", 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6423000000000001e-05 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400541+0000", 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0982999999999999e-05 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.890544020999997, 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011537824, 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.340 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 115 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:26.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:26.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4900000000000001e-07 2024-09-21T23:26:26.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389197+0000", 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5204999999999999e-05 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389203+0000", 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0979999999999999e-06 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389207+0000", 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6600000000000001e-06 2024-09-21T23:26:26.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395443+0000", 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062358099999999996 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395525+0000", 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1855999999999997e-05 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395669+0000", 2024-09-21T23:26:26.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014472399999999999 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397204+0000", 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001534817 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400599+0000", 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033944499999999998 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400655+0000", 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6647000000000002e-05 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400699+0000", 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4208000000000001e-05 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.890492569000003, 2024-09-21T23:26:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011571580999999999, 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 116 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3099999999999999e-07 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389216+0000", 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4260000000000002e-06 2024-09-21T23:26:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389223+0000", 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389225+0000", 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7760000000000002e-06 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396256+0000", 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070309020000000003 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396348+0000", 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1780000000000006e-05 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396481+0000", 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000133101 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397208+0000", 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072716199999999995 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400737+0000", 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035290220000000001 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400763+0000", 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5219999999999999e-05 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400785+0000", 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2045e-05 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.890475886000004, 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011687858000000001, 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.352 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 117 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3699999999999999e-07 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389232+0000", 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1789999999999998e-06 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389238+0000", 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209999999999999e-06 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389241+0000", 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.557e-06 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396792+0000", 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075506899999999997 2024-09-21T23:26:26.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396838+0000", 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5639000000000001e-05 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396958+0000", 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119892 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397458+0000", 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049987600000000001 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400824+0000", 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033662560000000002 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400880+0000", 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5779999999999998e-05 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400918+0000", 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8031000000000002e-05 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.878457412000003, 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006886924, 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.358 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 120 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:26.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401249+0000", 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.13e-07 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401286+0000", 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6773999999999997e-05 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401295+0000", 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9039999999999998e-06 2024-09-21T23:26:26.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401299+0000", 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3250000000000001e-06 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402183+0000", 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000883758 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402264+0000", 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1080000000000003e-05 2024-09-21T23:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402401+0000", 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013719700000000001 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403430+0000", 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010290119999999999 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408045+0000", 2024-09-21T23:26:26.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046146659999999999 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408095+0000", 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0898e-05 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408135+0000", 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9697000000000003e-05 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.878398938000004, 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069210920000000002, 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 121 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3200000000000001e-07 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401341+0000", 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4236999999999998e-05 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401346+0000", 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.04e-06 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401350+0000", 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6399999999999999e-06 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402733+0000", 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013825790000000001 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402816+0000", 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3461000000000006e-05 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402941+0000", 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012518300000000001 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403533+0000", 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000592071 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408174+0000", 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046404209999999996 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408204+0000", 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0147000000000001e-05 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408228+0000", 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3981e-05 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.442256040999993, 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064924400000000004, 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.369 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 139 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:26.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0800000000000001e-07 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837452+0000", 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7209999999999999e-06 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.371 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837455+0000", 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2279999999999998e-06 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837458+0000", 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8200000000000001e-06 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842187+0000", 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047298460000000002 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842233+0000", 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6056000000000001e-05 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842304+0000", 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0862e-05 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843291+0000", 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00098632100000000007 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843894+0000", 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060369199999999997 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843922+0000", 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7429000000000001e-05 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843942+0000", 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0256999999999999e-05 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.442244098000003, 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066155550000000004, 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 140 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1e-07 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837517+0000", 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5034000000000002e-05 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837522+0000", 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8520000000000003e-06 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837525+0000", 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4139999999999998e-06 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842562+0000", 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050366279999999996 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842615+0000", 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3675999999999999e-05 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842715+0000", 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9990999999999999e-05 2024-09-21T23:26:26.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843873+0000", 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011573 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844026+0000", 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015368900000000001 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844058+0000", 2024-09-21T23:26:26.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1449999999999999e-05 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844077+0000", 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9310999999999999e-05 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.442171904999995, 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068324850000000001, 2024-09-21T23:26:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 141 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.22e-07 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837539+0000", 2024-09-21T23:26:26.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.023e-06 2024-09-21T23:26:26.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.383 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837543+0000", 2024-09-21T23:26:26.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6490000000000001e-06 2024-09-21T23:26:26.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.384 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837546+0000", 2024-09-21T23:26:26.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.382e-06 2024-09-21T23:26:26.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.384 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843479+0000", 2024-09-21T23:26:26.385 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059328189999999998 2024-09-21T23:26:26.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.385 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.385 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843519+0000", 2024-09-21T23:26:26.385 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0071000000000003e-05 2024-09-21T23:26:26.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843600+0000", 2024-09-21T23:26:26.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1376000000000002e-05 2024-09-21T23:26:26.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844137+0000", 2024-09-21T23:26:26.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053640299999999997 2024-09-21T23:26:26.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844319+0000", 2024-09-21T23:26:26.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000181893 2024-09-21T23:26:26.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844346+0000", 2024-09-21T23:26:26.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.781e-05 2024-09-21T23:26:26.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844366+0000", 2024-09-21T23:26:26.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9737000000000001e-05 2024-09-21T23:26:26.388 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.389 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.389 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.389 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:26.389 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:26.390 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.826838209999998, 2024-09-21T23:26:26.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006594145, 2024-09-21T23:26:26.390 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.390 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.390 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.390 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.390 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.390 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.390 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 158 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452868+0000", 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9500000000000003e-07 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452938+0000", 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9598999999999996e-05 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452950+0000", 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2971e-05 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452960+0000", 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.448e-06 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456217+0000", 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003257378 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456557+0000", 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033989399999999998 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456703+0000", 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000145599 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459339+0000", 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026365529999999998 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459386+0000", 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6640000000000001e-05 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459428+0000", 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2148999999999999e-05 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459462+0000", 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3419e-05 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.826733836000002, 2024-09-21T23:26:26.395 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0065672760000000004, 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 159 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.396 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5400000000000001e-07 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452977+0000", 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.301e-06 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:26.397 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452984+0000", 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1840000000000002e-06 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452989+0000", 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5219999999999998e-06 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457324+0000", 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043348780000000003 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457416+0000", 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1824999999999996e-05 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457574+0000", 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015810999999999999 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459376+0000", 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001801649 2024-09-21T23:26:26.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459467+0000", 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1367999999999997e-05 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459507+0000", 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9576000000000002e-05 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459539+0000", 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2308999999999998e-05 2024-09-21T23:26:26.400 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.401 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.401 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.401 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:26.401 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:26.401 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:26.402 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:26.401+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:26.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:26.503 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:26.503+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:26.517 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:26.517 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:26.517 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:26.517 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:26.526 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:26.610 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:26.610+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:26.678 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:26.678 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:26.678 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:26.678 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:26.679 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:26.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:26.705 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:26.704+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:26.806 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:26.805+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:26.839 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.700906329999995, 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036127939999999999, 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3500000000000002e-07 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136608+0000", 2024-09-21T23:26:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.182e-06 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136618+0000", 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9199999999999999e-06 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136628+0000", 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4390000000000006e-06 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136692+0000", 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3979000000000005e-05 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136716+0000", 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4391000000000001e-05 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140171+0000", 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034554910000000002 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140214+0000", 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2057000000000001e-05 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'18)", 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.666388166999994, 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043957730000000004, 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171120+0000", 2024-09-21T23:26:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6700000000000003e-07 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171125+0000", 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.451e-06 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171138+0000", 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2541000000000001e-05 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171143+0000", 2024-09-21T23:26:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4659999999999998e-06 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171187+0000", 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4014000000000002e-05 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171200+0000", 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3108999999999999e-05 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175479+0000", 2024-09-21T23:26:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042784570000000003 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175515+0000", 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5967999999999999e-05 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:58 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.643635871000001, 2024-09-21T23:26:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003032851, 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193872+0000", 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9400000000000004e-07 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193952+0000", 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9791999999999998e-05 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193962+0000", 2024-09-21T23:26:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0034000000000001e-05 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193973+0000", 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0733e-05 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194015+0000", 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2911999999999998e-05 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194031+0000", 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5082999999999999e-05 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196869+0000", 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002838808 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196904+0000", 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4694999999999999e-05 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:59 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-21T23:26:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.643050204999994, 2024-09-21T23:26:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053675709999999998, 2024-09-21T23:26:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7100000000000002e-07 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194462+0000", 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.668e-06 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194469+0000", 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1670000000000002e-06 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194475+0000", 2024-09-21T23:26:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-06 2024-09-21T23:26:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194513+0000", 2024-09-21T23:26:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7391999999999997e-05 2024-09-21T23:26:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194531+0000", 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7870999999999999e-05 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199791+0000", 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052604119999999999 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199824+0000", 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3340000000000003e-05 2024-09-21T23:26:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:60 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'38, mlcod=16'36)", 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.63696951, 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003614562, 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200539+0000", 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5799999999999999e-06 2024-09-21T23:26:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200551+0000", 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1816e-05 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200578+0000", 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6772999999999999e-05 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200590+0000", 2024-09-21T23:26:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2303000000000001e-05 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200715+0000", 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012523499999999999 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200731+0000", 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5690000000000001e-05 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204121+0000", 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003390363 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204152+0000", 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0802000000000001e-05 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:70 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'39)", 2024-09-21T23:26:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.626913854999998, 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033896579999999998, 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210594+0000", 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1099999999999995e-07 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210600+0000", 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4389999999999997e-06 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210613+0000", 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.243e-05 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210618+0000", 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0939999999999998e-06 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210843+0000", 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022493200000000001 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210858+0000", 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5109000000000001e-05 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213938+0000", 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030796500000000002 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213983+0000", 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5293000000000002e-05 2024-09-21T23:26:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:72 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'46, mlcod=16'43)", 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.621969078999996, 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029102279999999999, 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215539+0000", 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4199999999999995e-07 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215589+0000", 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0488999999999997e-05 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215598+0000", 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2830000000000007e-06 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215610+0000", 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1134999999999999e-05 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215657+0000", 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7658999999999999e-05 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215679+0000", 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1526999999999999e-05 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218410+0000", 2024-09-21T23:26:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027316939999999998 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218448+0000", 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7799000000000002e-05 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:73 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'47, mlcod=16'43)", 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.621359732000002, 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051882020000000003, 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216148+0000", 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7100000000000001e-07 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216218+0000", 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0313000000000003e-05 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216227+0000", 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9959999999999999e-06 2024-09-21T23:26:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216232+0000", 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5340000000000001e-06 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216267+0000", 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4739000000000001e-05 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216281+0000", 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4345000000000001e-05 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221290+0000", 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050088989999999998 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221336+0000", 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5704999999999997e-05 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:74 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'48, mlcod=16'43)", 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.620833789000002, 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046922939999999996, 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216674+0000", 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9900000000000001e-07 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216679+0000", 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.502e-06 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216687+0000", 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1790000000000008e-06 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216691+0000", 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7230000000000002e-06 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216737+0000", 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6087000000000001e-05 2024-09-21T23:26:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216758+0000", 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0434000000000001e-05 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221338+0000", 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045804569999999996 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221366+0000", 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7413e-05 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:86 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'56, mlcod=16'50)", 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.603411855000004, 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049055269999999998, 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234096+0000", 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234099+0000", 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3890000000000001e-06 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234107+0000", 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2349999999999999e-06 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234112+0000", 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0629999999999999e-06 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234143+0000", 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1131999999999999e-05 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234155+0000", 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2140000000000001e-05 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.238972+0000", 2024-09-21T23:26:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048167770000000004 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.239001+0000", 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9164000000000001e-05 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:87 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'57, mlcod=16'52)", 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.60273248, 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067179910000000004, 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0299999999999998e-07 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234813+0000", 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7531999999999999e-05 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234826+0000", 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2617e-05 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234831+0000", 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1229999999999999e-06 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234869+0000", 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8513999999999998e-05 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234881+0000", 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2148e-05 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241465+0000", 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065837329999999996 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241493+0000", 2024-09-21T23:26:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7521e-05 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:88 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.601934631999995, 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059344530000000001, 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235573+0000", 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0100000000000004e-07 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235578+0000", 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1590000000000001e-06 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235590+0000", 2024-09-21T23:26:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1229e-05 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235595+0000", 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0889999999999999e-06 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235943+0000", 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000348227 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235957+0000", 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4419999999999999e-05 2024-09-21T23:26:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241494+0000", 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055371070000000003 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241507+0000", 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2520999999999999e-05 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:113 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'75, mlcod=16'71)", 2024-09-21T23:26:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.442703354000002, 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028543489999999999, 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8199999999999999e-07 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394809+0000", 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394821+0000", 2024-09-21T23:26:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2024999999999999e-05 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394827+0000", 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0020000000000003e-06 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394862+0000", 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4811000000000003e-05 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394873+0000", 2024-09-21T23:26:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.058e-05 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397631+0000", 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027581060000000002 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397658+0000", 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7073999999999999e-05 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'71)", 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.442291940999993, 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005160085, 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395216+0000", 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4999999999999998e-07 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395221+0000", 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2290000000000002e-06 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395229+0000", 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2369999999999992e-06 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395233+0000", 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1840000000000001e-06 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395260+0000", 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.728e-05 2024-09-21T23:26:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395272+0000", 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1270000000000001e-05 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400341+0000", 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050691809999999999 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400375+0000", 2024-09-21T23:26:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4254000000000001e-05 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'72)", 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.441743419999995, 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046274580000000001, 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395829+0000", 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4618999999999999e-05 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395836+0000", 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9090000000000003e-06 2024-09-21T23:26:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395839+0000", 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.371e-06 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395868+0000", 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9328000000000001e-05 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395880+0000", 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1501000000000001e-05 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400376+0000", 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044966340000000002 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400391+0000", 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4676e-05 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:116 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'78, mlcod=16'73)", 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.440938872999993, 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038400970000000002, 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396569+0000", 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1100000000000001e-07 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396573+0000", 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5569999999999997e-06 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396579+0000", 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1120000000000001e-06 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396587+0000", 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396630+0000", 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2991000000000002e-05 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396642+0000", 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1481000000000001e-05 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400392+0000", 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037507830000000002 2024-09-21T23:26:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400408+0000", 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5863e-05 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:117 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'79, mlcod=16'73)", 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.440436531000003, 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033512120000000001, 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7e-07 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397075+0000", 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7809999999999999e-06 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397082+0000", 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397085+0000", 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7139999999999999e-06 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397130+0000", 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.498e-05 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397142+0000", 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1838000000000001e-05 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400409+0000", 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032666349999999999 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400422+0000", 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2866e-05 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-09-21T23:26:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.435612857999999, 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034761240000000001, 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401895+0000", 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7400000000000003e-07 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401963+0000", 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8701000000000005e-05 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401972+0000", 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9530000000000005e-06 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401977+0000", 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1579999999999998e-06 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402009+0000", 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2512000000000003e-05 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402022+0000", 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3220999999999999e-05 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405337+0000", 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033150300000000001 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405370+0000", 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2975000000000002e-05 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.435006787999995, 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054168369999999999, 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402501+0000", 2024-09-21T23:26:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2699999999999999e-07 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402589+0000", 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899000000000007e-05 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402599+0000", 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0312000000000001e-05 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402605+0000", 2024-09-21T23:26:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9689999999999999e-06 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402648+0000", 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3210000000000001e-05 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402664+0000", 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5347999999999999e-05 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407887+0000", 2024-09-21T23:26:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052237919999999997 2024-09-21T23:26:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407917+0000", 2024-09-21T23:26:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9779999999999999e-05 2024-09-21T23:26:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 65.434477407000003, 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049049180000000003, 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4099999999999999e-07 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403098+0000", 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7464000000000003e-05 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403106+0000", 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6629999999999999e-06 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403111+0000", 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403137+0000", 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6349000000000001e-05 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403148+0000", 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.077e-05 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407920+0000", 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047716590000000001 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407935+0000", 2024-09-21T23:26:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4783999999999999e-05 2024-09-21T23:26:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:26.929 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:26.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T23:26:26.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:26.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:26.930+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:27.032 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:27.032+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:27.133 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:27.132+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:27.139 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:27.139 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:27.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:27.234 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:27.233+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:27.327 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:27.327 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:27.334 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:27.334+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:27.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:27.435 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:27.435+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:27.447 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-09-21T23:26:27.452 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 58.282112566999999, 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003527526, 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170743+0000", 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.08e-07 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170750+0000", 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800000000000001e-06 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170767+0000", 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6923999999999999e-05 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170788+0000", 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0446e-05 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170849+0000", 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1409999999999996e-05 2024-09-21T23:26:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170881+0000", 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2075e-05 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174218+0000", 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033371070000000002 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174270+0000", 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1375999999999997e-05 2024-09-21T23:26:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 57.266123073000003, 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011783570000000001, 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7999999999999996e-07 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186750+0000", 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7444000000000001e-05 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186834+0000", 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3974999999999995e-05 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186876+0000", 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2796000000000002e-05 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186999+0000", 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012303800000000001 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187036+0000", 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6897000000000003e-05 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187869+0000", 2024-09-21T23:26:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00083215199999999996 2024-09-21T23:26:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187910+0000", 2024-09-21T23:26:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1575000000000001e-05 2024-09-21T23:26:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:27.461 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:27.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:27.537 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:27.536+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:27.627 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:27.627 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:27.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:27.638 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:27.637+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:27.738 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:27.738+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:27.747 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:27.747 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:27.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:27.841 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:27.841+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:27.907 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:1 2.3 e19/18 2:c4c92e5a:::rbd_trash:head v 19'1, mlcod=0'0)", 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 59.739991899000003, 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039243489999999997, 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2999999999999999e-07 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167667+0000", 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7169999999999997e-06 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167679+0000", 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.276e-05 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167688+0000", 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8380000000000008e-06 2024-09-21T23:26:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167739+0000", 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0964999999999997e-05 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167762+0000", 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2221999999999998e-05 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171515+0000", 2024-09-21T23:26:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037531119999999999 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171582+0000", 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7504999999999998e-05 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4224.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:13.799177+0000", 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 14.108472748000001, 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00027751000000000002, 2024-09-21T23:26:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4224", 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.60:0/3116157232", 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 2 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799177+0000", 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799177+0000", 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799179+0000", 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.956e-06 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799196+0000", 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6646999999999998e-05 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799213+0000", 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7133e-05 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799225+0000", 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2044e-05 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799327+0000", 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000101983 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799454+0000", 2024-09-21T23:26:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000127747 2024-09-21T23:26:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:27.916 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:27.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:27.940 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:27.939+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:28.040 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:28.040+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:28.067 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:28.067 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:28.068 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:28.068 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:28.075 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:26:28.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:28.141 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:28.140+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:28.212 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:28.212 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:28.230 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:28.230 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:28.242 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:28.241+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:28.248 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:28.248 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:28.266 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:28.266 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:28.284 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:28.284 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:28.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:28.298 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:28.298 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:28.329 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:28.330 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:28.343 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:28.342+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:28.361 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:28.361 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:28.392 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:28.393 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:28.412 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:28.412 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:28.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:28.438 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:26:28.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:26:28.439 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T23:26:28.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T23:26:28.465 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:28.464+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:28.540 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:28.540 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:28.544 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:28.543+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:28.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:28.645 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:28.645+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:28.697 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:28.697 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:28.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:28.750 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:28.749+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:28.768 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:26:28.768 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:28.768 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 50, 2024-09-21T23:26:28.768 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "d51cc561-ddb8-41ef-b61c-7c6d698e4dd0", 2024-09-21T23:26:28.768 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-09-21T23:25:09.056809+0000", 2024-09-21T23:26:28.768 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-09-21T23:26:27.437444+0000", 2024-09-21T23:26:28.768 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-09-21T23:25:20.126996+0000", 2024-09-21T23:26:28.768 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-09-21T23:25:11.226810+0000", 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5806080, 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "nodeep-scrub", 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "noscrub", 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 5, 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T23:26:28.769 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 2, 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 8, 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "luminous", 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:19.818212+0000", 2024-09-21T23:26:28.770 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:28.771 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 1, 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 1, 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 1, 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 1, 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 1, 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.772 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "17", 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:28.773 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:28.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-09-21T23:26:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 6.9000000953674316, 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 6.9000000953674316, 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.28999999165534973, 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:28.776 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 2, 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "rbd", 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:26.968096+0000", 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 8193, 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:28.777 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 8, 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 8, 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 8, 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 8, 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 8, 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:28.778 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "21", 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 2, 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 21, 2024-09-21T23:26:28.779 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:28.780 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:28.781 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd": {} 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 1.75, 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 1.75, 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 1.1399999856948853, 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:26:28.782 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af", 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:28.783 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 35, 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:28.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6817/2339966024", 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6819/2339966024", 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6823/2339966024", 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6821/2339966024", 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:28.788 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "88ee06a3-9a6a-41e8-9327-a7a61304c03d", 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 49, 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:28.789 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.790 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.791 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.792 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.793 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6825/2889649822", 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6827/2889649822", 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6831/2889649822", 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6829/2889649822", 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:28.794 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05", 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 42, 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6809/864917651", 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6811/864917651", 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6815/864917651", 2024-09-21T23:26:28.799 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6813/864917651", 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "562e8c30-0622-4cb3-b5d1-f15b42a444bc", 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:28.800 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 48, 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.802 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-09-21T23:26:28.803 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.804 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6801/1713632981", 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6803/1713632981", 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6807/1713632981", 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6805/1713632981", 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7030ff68-4556-4ad5-abc9-be758f1caa85", 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:28.806 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6808", 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6809", 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:28.807 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6810", 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6811", 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.808 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6814", 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6815", 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.809 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6812", 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6813", 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6809/2660753894", 2024-09-21T23:26:28.810 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6811/2660753894", 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6815/2660753894", 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6813/2660753894", 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7169a163-f067-4385-b734-f58a3ba9e89e", 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:28.811 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6824", 2024-09-21T23:26:28.812 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6825", 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6826", 2024-09-21T23:26:28.813 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6827", 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6830", 2024-09-21T23:26:28.814 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6831", 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.815 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6828", 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6829", 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6825/2320687649", 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6827/2320687649", 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6831/2320687649", 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6829/2320687649", 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:28.816 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e511295d-2177-4dcb-a74b-b16b92534df6", 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:28.817 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6800", 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6801", 2024-09-21T23:26:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6802", 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6803", 2024-09-21T23:26:28.819 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6806", 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6807", 2024-09-21T23:26:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6804", 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6805", 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6801/3490482162", 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6803/3490482162", 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3490482162", 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6805/3490482162", 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "544f8a91-ce82-4855-85eb-ea814eb3d058", 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6816", 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6817", 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6818", 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6819", 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6822", 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6823", 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6820", 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6821", 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6817/1315554966", 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6819/1315554966", 2024-09-21T23:26:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6823/1315554966", 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6821/1315554966", 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:28.828 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.514872+0000", 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:28.829 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.912129+0000", 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.745154+0000", 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.830 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:16.758871+0000", 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:28.831 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.287462+0000", 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:18.031636+0000", 2024-09-21T23:26:28.832 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.055855+0000", 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.833 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.524609+0000", 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [ 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: 0, 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: 1 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "2.6", 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2, 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: 4 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [ 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "2.0", 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "mappings": [ 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "from": 1, 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "to": 7 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [], 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": {}, 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-09-21T23:26:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-09-21T23:26:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-09-21T23:26:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-09-21T23:26:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-09-21T23:26:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-09-21T23:26:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:28.838 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:28.839 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:28.839 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:28.840 INFO:tasks.thrashosds.thrasher:j is {'epoch': 50, 'fsid': 'd51cc561-ddb8-41ef-b61c-7c6d698e4dd0', 'created': '2024-09-21T23:25:09.056809+0000', 'modified': '2024-09-21T23:26:27.437444+0000', 'last_up_change': '2024-09-21T23:25:20.126996+0000', 'last_in_change': '2024-09-21T23:25:11.226810+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T23:25:19.818212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T23:25:26.968096+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 2339966024}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 2339966024}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 2339966024}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 2339966024}]}, 'public_addr': '172.21.15.31:6817/2339966024', 'cluster_addr': '172.21.15.31:6819/2339966024', 'heartbeat_back_addr': '172.21.15.31:6823/2339966024', 'heartbeat_front_addr': '172.21.15.31:6821/2339966024', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '88ee06a3-9a6a-41e8-9327-a7a61304c03d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 49, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 2889649822}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2889649822}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2889649822}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2889649822}]}, 'public_addr': '172.21.15.31:6825/2889649822', 'cluster_addr': '172.21.15.31:6827/2889649822', 'heartbeat_back_addr': '172.21.15.31:6831/2889649822', 'heartbeat_front_addr': '172.21.15.31:6829/2889649822', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 42, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 864917651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 864917651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 864917651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 864917651}]}, 'public_addr': '172.21.15.31:6809/864917651', 'cluster_addr': '172.21.15.31:6811/864917651', 'heartbeat_back_addr': '172.21.15.31:6815/864917651', 'heartbeat_front_addr': '172.21.15.31:6813/864917651', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '562e8c30-0622-4cb3-b5d1-f15b42a444bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6800', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6801', 'nonce': 1713632981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1713632981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1713632981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1713632981}]}, 'public_addr': '172.21.15.31:6801/1713632981', 'cluster_addr': '172.21.15.31:6803/1713632981', 'heartbeat_back_addr': '172.21.15.31:6807/1713632981', 'heartbeat_front_addr': '172.21.15.31:6805/1713632981', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7030ff68-4556-4ad5-abc9-be758f1caa85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6809', 'nonce': 2660753894}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6811', 'nonce': 2660753894}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6815', 'nonce': 2660753894}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6813', 'nonce': 2660753894}]}, 'public_addr': '172.21.15.60:6809/2660753894', 'cluster_addr': '172.21.15.60:6811/2660753894', 'heartbeat_back_addr': '172.21.15.60:6815/2660753894', 'heartbeat_front_addr': '172.21.15.60:6813/2660753894', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7169a163-f067-4385-b734-f58a3ba9e89e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6824', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6825', 'nonce': 2320687649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6826', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6827', 'nonce': 2320687649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6830', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6831', 'nonce': 2320687649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6828', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6829', 'nonce': 2320687649}]}, 'public_addr': '172.21.15.60:6825/2320687649', 'cluster_addr': '172.21.15.60:6827/2320687649', 'heartbeat_back_addr': '172.21.15.60:6831/2320687649', 'heartbeat_front_addr': '172.21.15.60:6829/2320687649', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e511295d-2177-4dcb-a74b-b16b92534df6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6801', 'nonce': 3490482162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6803', 'nonce': 3490482162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6807', 'nonce': 3490482162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6805', 'nonce': 3490482162}]}, 'public_addr': '172.21.15.60:6801/3490482162', 'cluster_addr': '172.21.15.60:6803/3490482162', 'heartbeat_back_addr': '172.21.15.60:6807/3490482162', 'heartbeat_front_addr': '172.21.15.60:6805/3490482162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '544f8a91-ce82-4855-85eb-ea814eb3d058', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6816', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6817', 'nonce': 1315554966}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6818', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6819', 'nonce': 1315554966}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6822', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6823', 'nonce': 1315554966}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6820', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6821', 'nonce': 1315554966}]}, 'public_addr': '172.21.15.60:6817/1315554966', 'cluster_addr': '172.21.15.60:6819/1315554966', 'heartbeat_back_addr': '172.21.15.60:6823/1315554966', 'heartbeat_front_addr': '172.21.15.60:6821/1315554966', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.514872+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.912129+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.745154+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:16.758871+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.287462+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:18.031636+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.055855+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.524609+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [0, 1]}, {'pgid': '2.6', 'osds': [2, 4]}], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 1, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T23:26:28.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:26:28.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:28.881 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:28.880+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:28.948 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:28.947+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:28.988 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:28.988 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:28.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:29.049 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:29.048+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:29.104 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:29.104 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:29.113 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:29.156 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:29.155+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:29.220 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:29.220 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:29.228 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:29.270 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:29.269+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:29.307 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:26:29.308 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:26:29.320 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-21T23:26:27.786010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":17,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237128,"kb_used_data":4216,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737592,"statfs":{"total":773094113280,"available":772851294208,"internally_reserved":0,"allocated":4317184,"data_stored":2891916,"data_compressed":19550,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000670"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":69,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-21T23:26:23.503594+0000","last_change":"2024-09-21T23:25:30.180125+0000","last_active":"2024-09-21T23:26:23.503594+0000","last_peered":"2024-09-21T23:26:23.503594+0000","last_clean":"2024-09-21T23:26:23.503594+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:26:23.503594+0000","last_undegraded":"2024-09-21T23:26:23.503594+0000","last_fullsized":"2024-09-21T23:26:23.503594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:17.515296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033759699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-21T23:26:25.569031+0000","last_change":"2024-09-21T23:26:17.546748+0000","last_active":"2024-09-21T23:26:25.569031+0000","last_peered":"2024-09-21T23:26:25.569031+0000","last_clean":"2024-09-21T23:26:25.569031+0000","last_became_active":"2024-09-21T23:26:17.546387+0000","last_became_peered":"2024-09-21T23:26:17.546387+0000","last_unstale":"2024-09-21T23:26:25.569031+0000","last_undegraded":"2024-09-21T23:26:25.569031+0000","last_fullsized":"2024-09-21T23:26:25.569031+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:56:12.695613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053117699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":72,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-21T23:26:25.517196+0000","last_change":"2024-09-21T23:25:30.263565+0000","last_active":"2024-09-21T23:26:25.517196+0000","last_peered":"2024-09-21T23:26:25.517196+0000","last_clean":"2024-09-21T23:26:25.517196+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:26:25.517196+0000","last_undegraded":"2024-09-21T23:26:25.517196+0000","last_fullsized":"2024-09-21T23:26:25.517196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:46:37.545348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042085499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":72,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-21T23:26:24.851953+0000","last_change":"2024-09-21T23:25:30.189121+0000","last_active":"2024-09-21T23:26:24.851953+0000","last_peered":"2024-09-21T23:26:24.851953+0000","last_clean":"2024-09-21T23:26:24.851953+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:26:24.851953+0000","last_undegraded":"2024-09-21T23:26:24.851953+0000","last_fullsized":"2024-09-21T23:26:24.851953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:00:22.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054857199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":73,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-21T23:26:25.569292+0000","last_change":"2024-09-21T23:25:30.186984+0000","last_active":"2024-09-21T23:26:25.569292+0000","last_peered":"2024-09-21T23:26:25.569292+0000","last_clean":"2024-09-21T23:26:25.569292+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:26:25.569292+0000","last_undegraded":"2024-09-21T23:26:25.569292+0000","last_fullsized":"2024-09-21T23:26:25.569292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:01:38.740017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095847300000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-21T23:26:25.569036+0000","last_change":"2024-09-21T23:25:30.263343+0000","last_active":"2024-09-21T23:26:25.569036+0000","last_peered":"2024-09-21T23:26:25.569036+0000","last_clean":"2024-09-21T23:26:25.569036+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:26:25.569036+0000","last_undegraded":"2024-09-21T23:26:25.569036+0000","last_fullsized":"2024-09-21T23:26:25.569036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:50:16.699815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044148899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":72,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-21T23:26:25.517194+0000","last_change":"2024-09-21T23:25:30.263501+0000","last_active":"2024-09-21T23:26:25.517194+0000","last_peered":"2024-09-21T23:26:25.517194+0000","last_clean":"2024-09-21T23:26:25.517194+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:26:25.517194+0000","last_undegraded":"2024-09-21T23:26:25.517194+0000","last_fullsized":"2024-09-21T23:26:25.517194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.155893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036082099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":72,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-21T23:26:25.569300+0000","last_change":"2024-09-21T23:25:30.186477+0000","last_active":"2024-09-21T23:26:25.569300+0000","last_peered":"2024-09-21T23:26:25.569300+0000","last_clean":"2024-09-21T23:26:25.569300+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:26:25.569300+0000","last_undegraded":"2024-09-21T23:26:25.569300+0000","last_fullsized":"2024-09-21T23:26:25.569300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:32:47.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044681799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":190,"reported_epoch":45,"state":"remapped+peering","last_fresh":"2024-09-21T23:26:23.418171+0000","last_change":"2024-09-21T23:26:23.850977+0000","last_active":"2024-09-21T23:26:23.850977+0000","last_peered":"2024-09-21T23:26:23.418171+0000","last_clean":"2024-09-21T23:26:23.418171+0000","last_became_active":"2024-09-21T23:26:01.315355+0000","last_became_peered":"2024-09-21T23:26:01.315355+0000","last_unstale":"2024-09-21T23:26:23.418171+0000","last_undegraded":"2024-09-21T23:26:23.418171+0000","last_fullsized":"2024-09-21T23:26:23.418171+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,4],"up_primary":1,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":47842,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":1,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542161,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":528384,"data_stored":213899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509458,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":532480,"data_stored":213918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542160,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31756,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340084,"statfs":{"total":96636764160,"available":96604246016,"internally_reserved":0,"allocated":581632,"data_stored":809829,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509457,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":528384,"data_stored":213899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509458,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27572,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":544768,"data_stored":219480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509457,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":528384,"data_stored":213899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":516096,"data_stored":208312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27572,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":557056,"data_stored":798680,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:26:29.321 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [3, 1, 2, 0] 2024-09-21T23:26:29.321 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.7', '3', '1', '2', '0'] 2024-09-21T23:26:29.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.7 3 1 2 0 2024-09-21T23:26:29.361 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:29.361+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:29.376 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:29.376 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:29.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:29.451 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:29.451+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:29.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T23:26:29.492 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:29.492 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:29.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:29.553 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:29.552+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:29.608 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:29.608 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:29.617 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:29.660 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:29.659+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:29.725 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:29.725 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:29.733 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:29.774 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:29.773+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:29.856 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:29.855+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:29.880 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:29.880 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:29.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:29.957 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:29.955+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:29.995 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:29.995 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:30.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:30.058 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:30.057+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:30.114 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:30.114 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:30.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:30.164 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:30.164+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:30.259 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:30.259+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:30.273 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:30.274 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:30.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:30.368 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:30.368+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:30.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:26:30.451 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:30.451 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:30.461 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:30.460+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:30.461 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-09-21T23:26:30.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:30.470 INFO:teuthology.orchestra.run.smithi031.stderr:set 2.7 pg_upmap_items mapping to [3->1,2->0] 2024-09-21T23:26:30.562 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:30.561+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:30.613 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:30.613 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:30.622 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:30.622 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:30.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:30.644 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:30.645 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:30.663 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:30.662+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:30.677 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:30.677 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:30.709 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:30.709 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:30.740 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:30.741 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:30.764 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:30.763+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:30.772 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:30.772 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:30.796 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:30.797 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:30.797 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:30.797 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:30.797 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:30.803 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:30.804 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:30.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:30.835 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:30.835 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:30.865 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:30.864+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:30.949 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:30.949 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:30.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 61.778812649000002, 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003527526, 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:30.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170743+0000", 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.08e-07 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170750+0000", 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800000000000001e-06 2024-09-21T23:26:30.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170767+0000", 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6923999999999999e-05 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170788+0000", 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0446e-05 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170849+0000", 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1409999999999996e-05 2024-09-21T23:26:30.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170881+0000", 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2075e-05 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174218+0000", 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033371070000000002 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174270+0000", 2024-09-21T23:26:30.953 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1375999999999997e-05 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 60.762823155, 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011783570000000001, 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:30.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7999999999999996e-07 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186750+0000", 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7444000000000001e-05 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186834+0000", 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3974999999999995e-05 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186876+0000", 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2796000000000002e-05 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.956 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186999+0000", 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012303800000000001 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187036+0000", 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6897000000000003e-05 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187869+0000", 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00083215199999999996 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:30.957 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187910+0000", 2024-09-21T23:26:30.958 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1575000000000001e-05 2024-09-21T23:26:30.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:30.958 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:30.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:30.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:30.958 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:30.958 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:30.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:31.002 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:31.002+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:31.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:31.067+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:31.069 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:31.070 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:31.070 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:31.070 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:31.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:31.169 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:31.168+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:31.228 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:31.229 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:31.229 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:31.229 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:31.229 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:31.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:31.279 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:31.279+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:31.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:31.370+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:31.388 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:31.389 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:31.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:31.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:31.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.389 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:1 2.3 e19/18 2:c4c92e5a:::rbd_trash:head v 19'1, mlcod=0'0)", 2024-09-21T23:26:31.389 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:31.389 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 63.221184690000001, 2024-09-21T23:26:31.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039243489999999997, 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:31.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:31.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:31.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:31.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2999999999999999e-07 2024-09-21T23:26:31.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167667+0000", 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7169999999999997e-06 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167679+0000", 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.276e-05 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167688+0000", 2024-09-21T23:26:31.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8380000000000008e-06 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167739+0000", 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0964999999999997e-05 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167762+0000", 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2221999999999998e-05 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.394 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:31.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171515+0000", 2024-09-21T23:26:31.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037531119999999999 2024-09-21T23:26:31.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.394 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:31.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171582+0000", 2024-09-21T23:26:31.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7504999999999998e-05 2024-09-21T23:26:31.394 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:31.394 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:31.394 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4224.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:13.799177+0000", 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 17.589665538999999, 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00027751000000000002, 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4224", 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.60:0/3116157232", 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 2 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.395 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799177+0000", 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799177+0000", 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799179+0000", 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.956e-06 2024-09-21T23:26:31.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799196+0000", 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6646999999999998e-05 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799213+0000", 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7133e-05 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799225+0000", 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2044e-05 2024-09-21T23:26:31.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799327+0000", 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000101983 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:13.799454+0000", 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000127747 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:31.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:31.399 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:31.399 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:31.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:31.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:31.484+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:31.572 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:31.571+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:31.610 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:31.610 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:31.611 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:31.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:31.672 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:31.672+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:31.772 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:31.772 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:31.772 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:31.772 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:31.772 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:31.773 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:31.773+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:31.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:31.874 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:31.874+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 32.589329266, 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00081488299999999997, 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:31.892 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301851+0000", 2024-09-21T23:26:31.893 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7899999999999998e-07 2024-09-21T23:26:31.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.893 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:31.893 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302047+0000", 2024-09-21T23:26:31.893 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019639200000000001 2024-09-21T23:26:31.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.893 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:31.893 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302064+0000", 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7380999999999998e-05 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302080+0000", 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5237e-05 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302153+0000", 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3922000000000001e-05 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.894 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302162+0000", 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8170000000000005e-06 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302665+0000", 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050245499999999996 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:31.895 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 32.586546048000002, 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00068581899999999995, 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:31.896 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304634+0000", 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5499999999999998e-07 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304796+0000", 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016237700000000001 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304813+0000", 2024-09-21T23:26:31.897 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.732e-05 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304825+0000", 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1302e-05 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304896+0000", 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1729000000000004e-05 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304905+0000", 2024-09-21T23:26:31.898 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8359999999999998e-06 2024-09-21T23:26:31.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:31.899 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:31.899 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:31.899 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.305319+0000", 2024-09-21T23:26:31.899 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00041360000000000002 2024-09-21T23:26:31.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:31.899 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:31.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:31.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:31.899 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:31.899 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:31.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:31.975 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:31.974+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:32.020 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:32.020 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:32.020 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:32.020 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:32.030 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:32.114 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:32.113+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:32.176 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:32.176+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:32.179 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:32.179 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:32.179 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:32.179 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:32.179 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:32.188 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:32.277 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:32.275+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:32.335 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:32.335 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:32.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:32.335 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:32.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.335 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:32.335 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:32.335 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 33.033638134999997, 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00108676, 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301959+0000", 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3099999999999996e-07 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302322+0000", 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00036268299999999998 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302345+0000", 2024-09-21T23:26:32.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3278e-05 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302367+0000", 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2118999999999999e-05 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302468+0000", 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100579 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302479+0000", 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1430999999999999e-05 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.303045+0000", 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000565739 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 33.030881897, 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010194869999999999, 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304715+0000", 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9599999999999998e-07 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305026+0000", 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031072100000000002 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:32.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305048+0000", 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1875999999999999e-05 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305066+0000", 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8644e-05 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305163+0000", 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6333000000000001e-05 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305172+0000", 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2189999999999993e-06 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305734+0000", 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056179799999999996 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.343 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:32.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:32.378 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:32.377+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:32.479 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:32.478+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:32.484 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T23:26:32.492 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:32.492 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:32.492 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:32.492 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:32.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:32.580 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:32.579+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:32.649 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:32.649 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:32.649 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:32.649 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:32.649 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:32.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:32.681 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:32.681+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:32.765 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:32.765 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:32.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:32.765 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:32.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.765 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:32.765 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:32.765 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.600767943999998, 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0080986310000000002, 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:32.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163709+0000", 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163727+0000", 2024-09-21T23:26:32.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7552000000000001e-05 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163744+0000", 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7541999999999999e-05 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163757+0000", 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2855e-05 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.768 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164069+0000", 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031179899999999997 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164084+0000", 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5265e-05 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167127+0000", 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030427029999999999 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.769 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167317+0000", 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000190006 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167465+0000", 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014824 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.168309+0000", 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084350899999999999 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171730+0000", 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034217620000000001 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171770+0000", 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0074999999999998e-05 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171802+0000", 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1563999999999998e-05 2024-09-21T23:26:32.771 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:1 2.3 e19/18 ondisk, result = 0)", 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.592825696999995, 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016281800000000001, 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6899999999999996e-07 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.773 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171651+0000", 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0309999999999998e-06 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171665+0000", 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.309e-05 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171672+0000", 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6070000000000002e-06 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171716+0000", 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4344000000000001e-05 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171727+0000", 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0488e-05 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171808+0000", 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1489000000000005e-05 2024-09-21T23:26:32.775 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.590050254999994, 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023803299999999999, 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:32.776 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 2 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174422+0000", 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4300000000000001e-06 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174429+0000", 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1659999999999999e-06 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174443+0000", 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4358e-05 2024-09-21T23:26:32.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174449+0000", 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2770000000000002e-06 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174540+0000", 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0209000000000003e-05 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174659+0000", 2024-09-21T23:26:32.779 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011959300000000001 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 64.589585255000003, 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016795100000000001, 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:32.780 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 3 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174887+0000", 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174892+0000", 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8180000000000003e-06 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:32.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174902+0000", 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0095000000000001e-05 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174911+0000", 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0799999999999995e-06 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174974+0000", 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3027000000000005e-05 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.175054+0000", 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0011000000000004e-05 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.596079623999998, 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006147263, 2024-09-21T23:26:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168394+0000", 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.956e-06 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168404+0000", 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0463e-05 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168423+0000", 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8715999999999998e-05 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168432+0000", 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.516e-06 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168529+0000", 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7073999999999999e-05 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168534+0000", 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.831e-06 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170224+0000", 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016893590000000001 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170332+0000", 2024-09-21T23:26:32.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108346 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170445+0000", 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112642 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.171210+0000", 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076543800000000001 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174469+0000", 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003259086 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174506+0000", 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6465999999999999e-05 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174539+0000", 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3370000000000001e-05 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.790 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:5 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.590137579, 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021298500000000001, 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:32.791 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174335+0000", 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.111e-06 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174341+0000", 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6239999999999996e-06 2024-09-21T23:26:32.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174356+0000", 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4548e-05 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174375+0000", 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8757e-05 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174457+0000", 2024-09-21T23:26:32.793 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2702999999999997e-05 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174466+0000", 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.208e-06 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174547+0000", 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1033999999999997e-05 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 62.580365540000003, 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040438610000000002, 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:32.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184109+0000", 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9069999999999999e-06 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184124+0000", 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5780000000000001e-05 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184159+0000", 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4106999999999997e-05 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184178+0000", 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9908999999999999e-05 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184330+0000", 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015152300000000001 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184337+0000", 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7920000000000004e-06 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186239+0000", 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019020129999999999 2024-09-21T23:26:32.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186307+0000", 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8326999999999998e-05 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186459+0000", 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152099 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187261+0000", 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00080186199999999997 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188085+0000", 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00082417199999999997 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188120+0000", 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5194999999999998e-05 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.800 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188150+0000", 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9175000000000001e-05 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:7 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 62.576494803999999, 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000179505, 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:32.801 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:32.802 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.16e-07 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187985+0000", 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3039999999999999e-06 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188000+0000", 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5097e-05 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188007+0000", 2024-09-21T23:26:32.803 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1330000000000001e-06 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188074+0000", 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7514000000000004e-05 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188082+0000", 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3649999999999998e-06 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188156+0000", 2024-09-21T23:26:32.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4176000000000003e-05 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4224.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.969018082000002, 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022720800000000001, 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4224", 2024-09-21T23:26:32.805 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/3116157232", 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795455+0000", 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0609999999999998e-06 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795473+0000", 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7762e-05 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795487+0000", 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.412e-05 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795496+0000", 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8780000000000004e-06 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795578+0000", 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1700999999999996e-05 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795680+0000", 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102686 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.808 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.809 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.809 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:32.809 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:32.809 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:32.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:32.811 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:32.809+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:32.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:26:32.911 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:32.910+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:32.917 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:32.917 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:32.917 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:32.918 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:32.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:32.979 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:32.980 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:33.009 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:33.009 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:33.011 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:33.011+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:33.039 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:33.039 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:33.056 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:33.056 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:33.072 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:33.073 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:33.075 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:33.075 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:33.075 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:33.075 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:33.075 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:33.084 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:33.089 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:33.089 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:33.105 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:33.106 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:33.122 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:33.122 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:33.128 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:33.127+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:33.191 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:33.191 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:33.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:33.212 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:33.212+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:33.312 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:33.311+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:33.347 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:33.347 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:33.356 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:33.439 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:33.438+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:33.467 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-09-21T23:26:33.504 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:33.505 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:33.513 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:33.529 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:33.529+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:33.616 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:33.615+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:33.661 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:33.661 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:33.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:33.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:33.716+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:33.782 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:33.782 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:33.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:33.817 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:33.817+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:33.918 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:33.917+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:34.000 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:34.001 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:34.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:34.057 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:34.057+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:34.120 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:34.119+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:34.182 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:34.182 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:34.194 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:34.221 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:34.220+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:34.321 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:34.321+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:34.354 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:34.355 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:34.364 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:34.422 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:34.421+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:34.523 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:34.522+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:34.533 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:34.533 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:34.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:34.627 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:34.627+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:34.692 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:34.692 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:34.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:34.742 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:34.742+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:34.825 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:34.825+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:34.853 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:34.853 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:34.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:34.926 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:34.926+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:34.972 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:34.972 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:34.981 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:35.065 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:35.065+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:35.127 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:35.127+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:35.134 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:35.134 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:35.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:26:35.144 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:35.228 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:35.228+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:35.304 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:35.304 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:35.320 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:35.320 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:35.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:35.330 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:35.329+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:35.334 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:35.334 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:35.366 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:35.367 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:35.398 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:35.399 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:35.430 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:35.429+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:35.430 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:35.430 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:35.440 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:35.441 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:35.441 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:35.441 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:35.441 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:35.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:35.462 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:35.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T23:26:35.493 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:26:35.494 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:26:35.494 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T23:26:35.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T23:26:35.494 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:35.495 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:35.528 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:35.528 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:35.531 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:35.530+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:35.621 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:35.621 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:35.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:35.621 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:35.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.621 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.319320067, 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00081488299999999997, 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301850+0000", 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.301851+0000", 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7899999999999998e-07 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302047+0000", 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00019639200000000001 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302064+0000", 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7380999999999998e-05 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302080+0000", 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5237e-05 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302153+0000", 2024-09-21T23:26:35.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3922000000000001e-05 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302162+0000", 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8170000000000005e-06 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.302665+0000", 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00050245499999999996 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 36.316536849000002, 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00068581899999999995, 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:35.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304633+0000", 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304634+0000", 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5499999999999998e-07 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304796+0000", 2024-09-21T23:26:35.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016237700000000001 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304813+0000", 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.732e-05 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304825+0000", 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1302e-05 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:35.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304896+0000", 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1729000000000004e-05 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.304905+0000", 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8359999999999998e-06 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:59.305319+0000", 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00041360000000000002 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.629 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.630 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.630 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:35.632 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:35.631+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:35.634 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:35.733 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:35.732+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:35.755 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:35.755 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:35.755 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:35.755 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:35.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:35.834 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:35.834+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:35.874 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:26:35.874 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:35.875 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 52, 2024-09-21T23:26:35.875 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "d51cc561-ddb8-41ef-b61c-7c6d698e4dd0", 2024-09-21T23:26:35.875 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-09-21T23:25:09.056809+0000", 2024-09-21T23:26:35.875 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-09-21T23:26:33.459126+0000", 2024-09-21T23:26:35.875 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-09-21T23:25:20.126996+0000", 2024-09-21T23:26:35.875 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-09-21T23:25:11.226810+0000", 2024-09-21T23:26:35.875 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T23:26:35.875 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5799936, 2024-09-21T23:26:35.875 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-09-21T23:26:35.875 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 5, 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 2, 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 8, 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "luminous", 2024-09-21T23:26:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:19.818212+0000", 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:35.877 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 1, 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 1, 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 1, 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 1, 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 1, 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:35.878 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "17", 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-09-21T23:26:35.879 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:35.880 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 6.9000000953674316, 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 6.9000000953674316, 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.28999999165534973, 2024-09-21T23:26:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 2, 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "rbd", 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:26.968096+0000", 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 8193, 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 8, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 8, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 8, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 8, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 8, 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "21", 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 2, 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 21, 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:35.886 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:35.887 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd": {} 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 1.75, 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 1.75, 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 1.1399999856948853, 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:26:35.888 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af", 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-09-21T23:26:35.889 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 35, 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.890 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-09-21T23:26:35.891 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:35.892 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6817/2339966024", 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6819/2339966024", 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6823/2339966024", 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6821/2339966024", 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:35.894 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "88ee06a3-9a6a-41e8-9327-a7a61304c03d", 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 49, 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.895 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:35.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6825/2889649822", 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6827/2889649822", 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6831/2889649822", 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6829/2889649822", 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:35.899 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05", 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 42, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:35.900 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6809/864917651", 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6811/864917651", 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6815/864917651", 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6813/864917651", 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "562e8c30-0622-4cb3-b5d1-f15b42a444bc", 2024-09-21T23:26:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 48, 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6801/1713632981", 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6803/1713632981", 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6807/1713632981", 2024-09-21T23:26:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6805/1713632981", 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7030ff68-4556-4ad5-abc9-be758f1caa85", 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6808", 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6809", 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6810", 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6811", 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6814", 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.914 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6815", 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6812", 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6813", 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:35.915 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6809/2660753894", 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6811/2660753894", 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6815/2660753894", 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6813/2660753894", 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7169a163-f067-4385-b734-f58a3ba9e89e", 2024-09-21T23:26:35.916 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6824", 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6825", 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6826", 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6827", 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6830", 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6831", 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6828", 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6829", 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6825/2320687649", 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6827/2320687649", 2024-09-21T23:26:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6831/2320687649", 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6829/2320687649", 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e511295d-2177-4dcb-a74b-b16b92534df6", 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6800", 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6801", 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6802", 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6803", 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6806", 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6807", 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6804", 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6805", 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6801/3490482162", 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6803/3490482162", 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3490482162", 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6805/3490482162", 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "544f8a91-ce82-4855-85eb-ea814eb3d058", 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:35.928 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6816", 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6817", 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6818", 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6819", 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6822", 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6823", 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.931 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6820", 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6821", 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.932 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6817/1315554966", 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6819/1315554966", 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6823/1315554966", 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6821/1315554966", 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-09-21T23:26:35.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.514872+0000", 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:35.934 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.912129+0000", 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.745154+0000", 2024-09-21T23:26:35.935 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:16.758871+0000", 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:35.936 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.287462+0000", 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:35.937 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:18.031636+0000", 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.055855+0000", 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:35.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.524609+0000", 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [ 2024-09-21T23:26:35.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: 0, 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: 1 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "2.6", 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: 2, 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: 4 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.940 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [ 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "2.0", 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: "mappings": [ 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: "from": 1, 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: "to": 7 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [], 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-09-21T23:26:35.941 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": {}, 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [], 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-09-21T23:26:35.942 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-09-21T23:26:35.943 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-09-21T23:26:35.943 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-09-21T23:26:35.943 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-09-21T23:26:35.943 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-09-21T23:26:35.943 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-09-21T23:26:35.943 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-09-21T23:26:35.943 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-09-21T23:26:35.943 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-09-21T23:26:35.943 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:35.943 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:35.945 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:35.945 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:35.946 INFO:tasks.thrashosds.thrasher:j is {'epoch': 52, 'fsid': 'd51cc561-ddb8-41ef-b61c-7c6d698e4dd0', 'created': '2024-09-21T23:25:09.056809+0000', 'modified': '2024-09-21T23:26:33.459126+0000', 'last_up_change': '2024-09-21T23:25:20.126996+0000', 'last_in_change': '2024-09-21T23:25:11.226810+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T23:25:19.818212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T23:25:26.968096+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 2339966024}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 2339966024}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 2339966024}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 2339966024}]}, 'public_addr': '172.21.15.31:6817/2339966024', 'cluster_addr': '172.21.15.31:6819/2339966024', 'heartbeat_back_addr': '172.21.15.31:6823/2339966024', 'heartbeat_front_addr': '172.21.15.31:6821/2339966024', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '88ee06a3-9a6a-41e8-9327-a7a61304c03d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 49, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 2889649822}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2889649822}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2889649822}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2889649822}]}, 'public_addr': '172.21.15.31:6825/2889649822', 'cluster_addr': '172.21.15.31:6827/2889649822', 'heartbeat_back_addr': '172.21.15.31:6831/2889649822', 'heartbeat_front_addr': '172.21.15.31:6829/2889649822', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 42, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 864917651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 864917651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 864917651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 864917651}]}, 'public_addr': '172.21.15.31:6809/864917651', 'cluster_addr': '172.21.15.31:6811/864917651', 'heartbeat_back_addr': '172.21.15.31:6815/864917651', 'heartbeat_front_addr': '172.21.15.31:6813/864917651', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '562e8c30-0622-4cb3-b5d1-f15b42a444bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6800', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6801', 'nonce': 1713632981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1713632981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1713632981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1713632981}]}, 'public_addr': '172.21.15.31:6801/1713632981', 'cluster_addr': '172.21.15.31:6803/1713632981', 'heartbeat_back_addr': '172.21.15.31:6807/1713632981', 'heartbeat_front_addr': '172.21.15.31:6805/1713632981', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7030ff68-4556-4ad5-abc9-be758f1caa85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6809', 'nonce': 2660753894}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6811', 'nonce': 2660753894}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6815', 'nonce': 2660753894}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6813', 'nonce': 2660753894}]}, 'public_addr': '172.21.15.60:6809/2660753894', 'cluster_addr': '172.21.15.60:6811/2660753894', 'heartbeat_back_addr': '172.21.15.60:6815/2660753894', 'heartbeat_front_addr': '172.21.15.60:6813/2660753894', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7169a163-f067-4385-b734-f58a3ba9e89e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6824', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6825', 'nonce': 2320687649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6826', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6827', 'nonce': 2320687649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6830', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6831', 'nonce': 2320687649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6828', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6829', 'nonce': 2320687649}]}, 'public_addr': '172.21.15.60:6825/2320687649', 'cluster_addr': '172.21.15.60:6827/2320687649', 'heartbeat_back_addr': '172.21.15.60:6831/2320687649', 'heartbeat_front_addr': '172.21.15.60:6829/2320687649', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e511295d-2177-4dcb-a74b-b16b92534df6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6801', 'nonce': 3490482162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6803', 'nonce': 3490482162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6807', 'nonce': 3490482162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6805', 'nonce': 3490482162}]}, 'public_addr': '172.21.15.60:6801/3490482162', 'cluster_addr': '172.21.15.60:6803/3490482162', 'heartbeat_back_addr': '172.21.15.60:6807/3490482162', 'heartbeat_front_addr': '172.21.15.60:6805/3490482162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '544f8a91-ce82-4855-85eb-ea814eb3d058', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6816', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6817', 'nonce': 1315554966}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6818', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6819', 'nonce': 1315554966}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6822', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6823', 'nonce': 1315554966}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6820', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6821', 'nonce': 1315554966}]}, 'public_addr': '172.21.15.60:6817/1315554966', 'cluster_addr': '172.21.15.60:6819/1315554966', 'heartbeat_back_addr': '172.21.15.60:6823/1315554966', 'heartbeat_front_addr': '172.21.15.60:6821/1315554966', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.514872+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.912129+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.745154+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:16.758871+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.287462+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:18.031636+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.055855+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.524609+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [0, 1]}, {'pgid': '2.6', 'osds': [2, 4]}], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 1, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T23:26:35.946 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.0 2024-09-21T23:26:35.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap-items 2.0 2024-09-21T23:26:35.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:35.944+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:35.947 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:36.045 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:36.045+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:36.053 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 36.752147788000002, 2024-09-21T23:26:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00108676, 2024-09-21T23:26:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301959+0000", 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3099999999999996e-07 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302322+0000", 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00036268299999999998 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302345+0000", 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3278e-05 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302367+0000", 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2118999999999999e-05 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302468+0000", 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100579 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302479+0000", 2024-09-21T23:26:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1430999999999999e-05 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.303045+0000", 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000565739 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 36.749391549999999, 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010194869999999999, 2024-09-21T23:26:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304715+0000", 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9599999999999998e-07 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305026+0000", 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031072100000000002 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305048+0000", 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1875999999999999e-05 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305066+0000", 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8644e-05 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305163+0000", 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6333000000000001e-05 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305172+0000", 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2189999999999993e-06 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305734+0000", 2024-09-21T23:26:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056179799999999996 2024-09-21T23:26:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.062 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:36.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:36.146 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:36.145+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:36.170 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:36.170 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:36.171 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:36.171 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:36.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:36.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:36.247+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:36.326 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:36.326 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:36.326 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:36.326 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:36.326 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:36.335 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:36.378 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:36.377+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:36.442 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:36.442 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:36.442 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 68.278337296999993, 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0080986310000000002, 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:36.443 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163709+0000", 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163727+0000", 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7552000000000001e-05 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163744+0000", 2024-09-21T23:26:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7541999999999999e-05 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163757+0000", 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2855e-05 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164069+0000", 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031179899999999997 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164084+0000", 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5265e-05 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167127+0000", 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030427029999999999 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167317+0000", 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000190006 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167465+0000", 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014824 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.168309+0000", 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084350899999999999 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171730+0000", 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034217620000000001 2024-09-21T23:26:36.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171770+0000", 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0074999999999998e-05 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171802+0000", 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1563999999999998e-05 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:1 2.3 e19/18 ondisk, result = 0)", 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 68.270395050000005, 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016281800000000001, 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6899999999999996e-07 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:36.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171651+0000", 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0309999999999998e-06 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171665+0000", 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.309e-05 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171672+0000", 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6070000000000002e-06 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171716+0000", 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4344000000000001e-05 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171727+0000", 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0488e-05 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171808+0000", 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1489000000000005e-05 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.453 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 68.267619608000004, 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023803299999999999, 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:36.454 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 2 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174422+0000", 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4300000000000001e-06 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174429+0000", 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1659999999999999e-06 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:36.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174443+0000", 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4358e-05 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174449+0000", 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2770000000000002e-06 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174540+0000", 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0209000000000003e-05 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174659+0000", 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011959300000000001 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 68.267154607999998, 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016795100000000001, 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:36.458 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 3 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174887+0000", 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174892+0000", 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8180000000000003e-06 2024-09-21T23:26:36.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174902+0000", 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0095000000000001e-05 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174911+0000", 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0799999999999995e-06 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174974+0000", 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3027000000000005e-05 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.175054+0000", 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0011000000000004e-05 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 67.273648976999993, 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006147263, 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168394+0000", 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.956e-06 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168404+0000", 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0463e-05 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168423+0000", 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8715999999999998e-05 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168432+0000", 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.516e-06 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.465 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168529+0000", 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7073999999999999e-05 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168534+0000", 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.831e-06 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170224+0000", 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016893590000000001 2024-09-21T23:26:36.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170332+0000", 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108346 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170445+0000", 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112642 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:36.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.171210+0000", 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076543800000000001 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174469+0000", 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003259086 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174506+0000", 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6465999999999999e-05 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174539+0000", 2024-09-21T23:26:36.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3370000000000001e-05 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:5 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 67.267706931999996, 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021298500000000001, 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:36.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174335+0000", 2024-09-21T23:26:36.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.111e-06 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174341+0000", 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6239999999999996e-06 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174356+0000", 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4548e-05 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:36.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174375+0000", 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8757e-05 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174457+0000", 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2702999999999997e-05 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174466+0000", 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.208e-06 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174547+0000", 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1033999999999997e-05 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 66.257934892999998, 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040438610000000002, 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:36.473 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184109+0000", 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9069999999999999e-06 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184124+0000", 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5780000000000001e-05 2024-09-21T23:26:36.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184159+0000", 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4106999999999997e-05 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184178+0000", 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9908999999999999e-05 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.476 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184330+0000", 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015152300000000001 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184337+0000", 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7920000000000004e-06 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186239+0000", 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019020129999999999 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186307+0000", 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8326999999999998e-05 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186459+0000", 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152099 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187261+0000", 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00080186199999999997 2024-09-21T23:26:36.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188085+0000", 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00082417199999999997 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188120+0000", 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5194999999999998e-05 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188150+0000", 2024-09-21T23:26:36.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9175000000000001e-05 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:7 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 66.254064157000002, 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000179505, 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.16e-07 2024-09-21T23:26:36.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187985+0000", 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3039999999999999e-06 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188000+0000", 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5097e-05 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188007+0000", 2024-09-21T23:26:36.482 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1330000000000001e-06 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188074+0000", 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7514000000000004e-05 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188082+0000", 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3649999999999998e-06 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:36.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188156+0000", 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4176000000000003e-05 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4224.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.646587435000001, 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022720800000000001, 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:36.484 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4224", 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/3116157232", 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:36.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795455+0000", 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0609999999999998e-06 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795473+0000", 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7762e-05 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795487+0000", 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.412e-05 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795496+0000", 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8780000000000004e-06 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795578+0000", 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1700999999999996e-05 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:36.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:36.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795680+0000", 2024-09-21T23:26:36.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102686 2024-09-21T23:26:36.488 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.488 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.488 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.488 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:36.488 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:36.488 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:36.489 INFO:teuthology.orchestra.run.smithi031.stderr:clear 2.0 pg_upmap_items mapping 2024-09-21T23:26:36.489 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-09-21T23:26:36.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:36.490 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:36.488+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:36.590 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:36.589+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:36.637 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:36.637 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:36.638 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:36.638 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:36.646 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:36.691 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:36.689+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:36.754 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:36.754 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:36.754 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:36.754 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:36.754 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:36.762 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:36.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:36.791+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:36.894 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:36.891+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:36.910 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:36.911 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:36.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:36.911 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-09-21T23:26:36.911 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:36.919 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:37.004 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:37.003+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:37.067 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:37.067 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:37.067 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:37.067 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:37.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:37.095 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:37.095+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:37.196 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:37.195+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:37.223 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:37.223 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:37.224 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:37.224 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:37.224 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:37.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:37.316 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:37.315+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:37.382 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.382 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 76.213235882000006, 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090066050000000009, 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 37 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7099999999999998e-07 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167009+0000", 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.92e-06 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167013+0000", 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.385 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167017+0000", 2024-09-21T23:26:37.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7280000000000001e-06 2024-09-21T23:26:37.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170778+0000", 2024-09-21T23:26:37.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037612399999999999 2024-09-21T23:26:37.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170831+0000", 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2694000000000001e-05 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170970+0000", 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000138966 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.171555+0000", 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058538399999999999 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175872+0000", 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043172389999999996 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175969+0000", 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6844999999999999e-05 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.176012+0000", 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3356999999999998e-05 2024-09-21T23:26:37.388 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 76.187131046000005, 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069023069999999999, 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.389 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0999999999999999e-07 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193115+0000", 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5700000000000001e-06 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193129+0000", 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4124000000000001e-05 2024-09-21T23:26:37.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193133+0000", 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0550000000000001e-06 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194144+0000", 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001010735 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194231+0000", 2024-09-21T23:26:37.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7665999999999997e-05 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194357+0000", 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125353 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194844+0000", 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000487305 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199964+0000", 2024-09-21T23:26:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051204859999999996 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199992+0000", 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7902000000000002e-05 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.200013+0000", 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0601000000000001e-05 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.394 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.395 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.395 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:37.395 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 76.165481971999995, 2024-09-21T23:26:37.395 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0067494620000000003, 2024-09-21T23:26:37.395 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.395 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.395 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.395 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.396 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.396 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.396 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 73 2024-09-21T23:26:37.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.396 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.396 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.397 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:37.397 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.397 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.397 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:37.397 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.398 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.398 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:37.398 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4900000000000001e-07 2024-09-21T23:26:37.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.399 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.399 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214818+0000", 2024-09-21T23:26:37.399 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7754000000000003e-05 2024-09-21T23:26:37.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.399 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.400 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214842+0000", 2024-09-21T23:26:37.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4552000000000001e-05 2024-09-21T23:26:37.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.400 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.400 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214846+0000", 2024-09-21T23:26:37.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9129999999999996e-06 2024-09-21T23:26:37.401 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.401 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.401 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.401 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215813+0000", 2024-09-21T23:26:37.401 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00096695899999999998 2024-09-21T23:26:37.401 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.402 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.402 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.402 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215904+0000", 2024-09-21T23:26:37.402 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0328000000000007e-05 2024-09-21T23:26:37.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.402 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.402 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.403 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216020+0000", 2024-09-21T23:26:37.403 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115838 2024-09-21T23:26:37.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.403 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.403 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.404 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217018+0000", 2024-09-21T23:26:37.404 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099836200000000003 2024-09-21T23:26:37.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.404 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.404 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.404 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221461+0000", 2024-09-21T23:26:37.404 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044435969999999997 2024-09-21T23:26:37.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.405 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.405 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.405 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221487+0000", 2024-09-21T23:26:37.405 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5806e-05 2024-09-21T23:26:37.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.406 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.406 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221509+0000", 2024-09-21T23:26:37.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1903999999999999e-05 2024-09-21T23:26:37.406 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.406 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.406 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.407 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.407 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.407 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:37.407 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 76.165387269000007, 2024-09-21T23:26:37.407 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068139230000000004, 2024-09-21T23:26:37.407 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.408 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.408 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.408 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.408 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.408 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.408 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 74 2024-09-21T23:26:37.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.409 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.409 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.409 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:37.409 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.410 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.410 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:37.410 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.410 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214855+0000", 2024-09-21T23:26:37.411 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-07 2024-09-21T23:26:37.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.411 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214859+0000", 2024-09-21T23:26:37.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1389999999999997e-06 2024-09-21T23:26:37.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.412 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.412 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214871+0000", 2024-09-21T23:26:37.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1698999999999999e-05 2024-09-21T23:26:37.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.413 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.413 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214887+0000", 2024-09-21T23:26:37.413 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6061000000000001e-05 2024-09-21T23:26:37.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.414 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.414 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216359+0000", 2024-09-21T23:26:37.414 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014720779999999999 2024-09-21T23:26:37.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.414 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.415 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216439+0000", 2024-09-21T23:26:37.415 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0191999999999994e-05 2024-09-21T23:26:37.415 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.415 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.415 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.415 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216579+0000", 2024-09-21T23:26:37.415 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014029099999999999 2024-09-21T23:26:37.416 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.416 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.416 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217129+0000", 2024-09-21T23:26:37.416 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054932099999999997 2024-09-21T23:26:37.416 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.417 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.417 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221559+0000", 2024-09-21T23:26:37.417 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044299710000000004 2024-09-21T23:26:37.417 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.417 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.417 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.418 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221627+0000", 2024-09-21T23:26:37.418 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8631000000000001e-05 2024-09-21T23:26:37.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.418 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.418 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221668+0000", 2024-09-21T23:26:37.419 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0988000000000001e-05 2024-09-21T23:26:37.419 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.419 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.419 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.420 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.420 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:37.420 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 76.148711082999995, 2024-09-21T23:26:37.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0077112509999999997, 2024-09-21T23:26:37.420 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.420 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.421 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.421 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.421 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.421 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.421 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 86 2024-09-21T23:26:37.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.422 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.422 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:37.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.423 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:37.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:37.424 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5199999999999998e-07 2024-09-21T23:26:37.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.424 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.424 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231533+0000", 2024-09-21T23:26:37.424 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3879999999999998e-06 2024-09-21T23:26:37.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.425 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.425 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.425 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231537+0000", 2024-09-21T23:26:37.425 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0010000000000003e-06 2024-09-21T23:26:37.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231540+0000", 2024-09-21T23:26:37.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0489999999999999e-06 2024-09-21T23:26:37.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.427 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233890+0000", 2024-09-21T23:26:37.427 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023500249999999999 2024-09-21T23:26:37.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.427 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.427 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233937+0000", 2024-09-21T23:26:37.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6400999999999998e-05 2024-09-21T23:26:37.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.428 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.428 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234015+0000", 2024-09-21T23:26:37.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8343000000000005e-05 2024-09-21T23:26:37.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.429 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.429 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235675+0000", 2024-09-21T23:26:37.429 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001660124 2024-09-21T23:26:37.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.430 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.430 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239138+0000", 2024-09-21T23:26:37.430 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034624109999999999 2024-09-21T23:26:37.430 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.430 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.431 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239200+0000", 2024-09-21T23:26:37.431 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2698000000000002e-05 2024-09-21T23:26:37.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.431 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.431 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239242+0000", 2024-09-21T23:26:37.431 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1458999999999997e-05 2024-09-21T23:26:37.432 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.432 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.432 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.432 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.432 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:37.432 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 76.148694813999995, 2024-09-21T23:26:37.432 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010200097999999999, 2024-09-21T23:26:37.432 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.433 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.433 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.433 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.433 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.433 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.433 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87 2024-09-21T23:26:37.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.433 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.434 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:37.434 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.434 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.434 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.434 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:37.434 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.434 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.434 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.435 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:37.435 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2700000000000001e-07 2024-09-21T23:26:37.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.435 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.435 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.435 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231588+0000", 2024-09-21T23:26:37.435 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0985000000000001e-05 2024-09-21T23:26:37.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.435 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.435 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.436 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231593+0000", 2024-09-21T23:26:37.436 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1059999999999999e-06 2024-09-21T23:26:37.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.436 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.436 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.436 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231606+0000", 2024-09-21T23:26:37.436 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2343e-05 2024-09-21T23:26:37.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.436 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.436 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.437 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234406+0000", 2024-09-21T23:26:37.437 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027999850000000001 2024-09-21T23:26:37.437 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.437 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.437 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.437 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234489+0000", 2024-09-21T23:26:37.438 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3051000000000002e-05 2024-09-21T23:26:37.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.438 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.438 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.438 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234631+0000", 2024-09-21T23:26:37.438 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285799999999999 2024-09-21T23:26:37.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.438 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.438 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.439 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235679+0000", 2024-09-21T23:26:37.439 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001047517 2024-09-21T23:26:37.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.439 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.439 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.439 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241644+0000", 2024-09-21T23:26:37.439 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059652129999999996 2024-09-21T23:26:37.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.439 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.439 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.440 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241704+0000", 2024-09-21T23:26:37.440 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9732999999999999e-05 2024-09-21T23:26:37.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.440 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.440 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.440 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241747+0000", 2024-09-21T23:26:37.440 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-05 2024-09-21T23:26:37.440 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.440 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.440 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.441 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.441 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.441 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:37.441 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 76.148628974999994, 2024-09-21T23:26:37.441 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010246534, 2024-09-21T23:26:37.441 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.441 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.441 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 88 2024-09-21T23:26:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.443 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.443 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:37.443 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.444 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:37.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0199999999999998e-07 2024-09-21T23:26:37.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.444 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231615+0000", 2024-09-21T23:26:37.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3439999999999999e-06 2024-09-21T23:26:37.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.445 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231619+0000", 2024-09-21T23:26:37.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0219999999999998e-06 2024-09-21T23:26:37.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231622+0000", 2024-09-21T23:26:37.446 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6989999999999999e-06 2024-09-21T23:26:37.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235337+0000", 2024-09-21T23:26:37.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037150030000000001 2024-09-21T23:26:37.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235385+0000", 2024-09-21T23:26:37.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7861999999999997e-05 2024-09-21T23:26:37.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235458+0000", 2024-09-21T23:26:37.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2596999999999996e-05 2024-09-21T23:26:37.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235978+0000", 2024-09-21T23:26:37.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052013600000000004 2024-09-21T23:26:37.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.450 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.450 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.450 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241799+0000", 2024-09-21T23:26:37.450 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058215289999999998 2024-09-21T23:26:37.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.450 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.450 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241836+0000", 2024-09-21T23:26:37.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6282999999999998e-05 2024-09-21T23:26:37.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241859+0000", 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3757e-05 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 75.991237792000007, 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072206969999999999, 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.452 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 112 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:37.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.72e-07 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389069+0000", 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4875000000000005e-05 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.454 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389079+0000", 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0135e-05 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389085+0000", 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394006+0000", 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049207920000000002 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394052+0000", 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5810999999999999e-05 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394139+0000", 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7130000000000001e-05 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395407+0000", 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012681260000000001 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396139+0000", 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000731703 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396194+0000", 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5309999999999997e-05 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396225+0000", 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0589000000000001e-05 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.457 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 75.991130835000007, 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086923490000000003, 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 113 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.458 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.459 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389116+0000", 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.245e-06 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389126+0000", 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0295000000000001e-05 2024-09-21T23:26:37.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389132+0000", 2024-09-21T23:26:37.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0889999999999996e-06 2024-09-21T23:26:37.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394548+0000", 2024-09-21T23:26:37.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005415876 2024-09-21T23:26:37.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.462 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394642+0000", 2024-09-21T23:26:37.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4531999999999995e-05 2024-09-21T23:26:37.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.462 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394714+0000", 2024-09-21T23:26:37.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1753000000000002e-05 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396233+0000", 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015185299999999999 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397749+0000", 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015157770000000001 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397779+0000", 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0561000000000001e-05 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397803+0000", 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4116000000000001e-05 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.464 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 75.991098765999993, 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.01139824, 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 114 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:37.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.01e-07 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389146+0000", 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5239999999999999e-06 2024-09-21T23:26:37.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389150+0000", 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3989999999999997e-06 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389153+0000", 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.038e-06 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395026+0000", 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058731720000000003 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395071+0000", 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4388999999999998e-05 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395141+0000", 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0053000000000002e-05 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397196+0000", 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020555899999999999 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400494+0000", 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032972679999999999 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400520+0000", 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6423000000000001e-05 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400541+0000", 2024-09-21T23:26:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0982999999999999e-05 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 75.991080087, 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011537824, 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 115 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4900000000000001e-07 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389197+0000", 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5204999999999999e-05 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389203+0000", 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0979999999999999e-06 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389207+0000", 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6600000000000001e-06 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395443+0000", 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062358099999999996 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395525+0000", 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1855999999999997e-05 2024-09-21T23:26:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395669+0000", 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014472399999999999 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397204+0000", 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001534817 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400599+0000", 2024-09-21T23:26:37.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033944499999999998 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400655+0000", 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6647000000000002e-05 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400699+0000", 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4208000000000001e-05 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 75.991028635000006, 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011571580999999999, 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 116 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.476 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3099999999999999e-07 2024-09-21T23:26:37.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389216+0000", 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4260000000000002e-06 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389223+0000", 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389225+0000", 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7760000000000002e-06 2024-09-21T23:26:37.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396256+0000", 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070309020000000003 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396348+0000", 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1780000000000006e-05 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396481+0000", 2024-09-21T23:26:37.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000133101 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397208+0000", 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072716199999999995 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400737+0000", 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035290220000000001 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400763+0000", 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5219999999999999e-05 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400785+0000", 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2045e-05 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:37.481 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 75.991011951999994, 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011687858000000001, 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 117 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3699999999999999e-07 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389232+0000", 2024-09-21T23:26:37.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1789999999999998e-06 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389238+0000", 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209999999999999e-06 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389241+0000", 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.557e-06 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396792+0000", 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075506899999999997 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396838+0000", 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5639000000000001e-05 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396958+0000", 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119892 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397458+0000", 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049987600000000001 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400824+0000", 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033662560000000002 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400880+0000", 2024-09-21T23:26:37.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5779999999999998e-05 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400918+0000", 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8031000000000002e-05 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.487 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 75.978993478000007, 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006886924, 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 120 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401249+0000", 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.13e-07 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401286+0000", 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6773999999999997e-05 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401295+0000", 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9039999999999998e-06 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401299+0000", 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3250000000000001e-06 2024-09-21T23:26:37.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402183+0000", 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000883758 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402264+0000", 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1080000000000003e-05 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402401+0000", 2024-09-21T23:26:37.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013719700000000001 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403430+0000", 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010290119999999999 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408045+0000", 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046146659999999999 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408095+0000", 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0898e-05 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408135+0000", 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9697000000000003e-05 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.493 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 75.978935003999993, 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069210920000000002, 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 121 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3200000000000001e-07 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401341+0000", 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4236999999999998e-05 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401346+0000", 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.04e-06 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401350+0000", 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6399999999999999e-06 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402733+0000", 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013825790000000001 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402816+0000", 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3461000000000006e-05 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402941+0000", 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012518300000000001 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403533+0000", 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000592071 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408174+0000", 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046404209999999996 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408204+0000", 2024-09-21T23:26:37.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0147000000000001e-05 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408228+0000", 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3981e-05 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 75.542792106999997, 2024-09-21T23:26:37.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064924400000000004, 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 139 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.501 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.501 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:37.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.501 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.501 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:37.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0800000000000001e-07 2024-09-21T23:26:37.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.502 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.502 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837452+0000", 2024-09-21T23:26:37.502 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7209999999999999e-06 2024-09-21T23:26:37.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.502 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837455+0000", 2024-09-21T23:26:37.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2279999999999998e-06 2024-09-21T23:26:37.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837458+0000", 2024-09-21T23:26:37.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8200000000000001e-06 2024-09-21T23:26:37.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842187+0000", 2024-09-21T23:26:37.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047298460000000002 2024-09-21T23:26:37.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842233+0000", 2024-09-21T23:26:37.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6056000000000001e-05 2024-09-21T23:26:37.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842304+0000", 2024-09-21T23:26:37.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0862e-05 2024-09-21T23:26:37.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843291+0000", 2024-09-21T23:26:37.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00098632100000000007 2024-09-21T23:26:37.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843894+0000", 2024-09-21T23:26:37.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060369199999999997 2024-09-21T23:26:37.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843922+0000", 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7429000000000001e-05 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843942+0000", 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0256999999999999e-05 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 75.542780164000007, 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066155550000000004, 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 140 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1e-07 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837517+0000", 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5034000000000002e-05 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837522+0000", 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8520000000000003e-06 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837525+0000", 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4139999999999998e-06 2024-09-21T23:26:37.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842562+0000", 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050366279999999996 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842615+0000", 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3675999999999999e-05 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842715+0000", 2024-09-21T23:26:37.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9990999999999999e-05 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843873+0000", 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011573 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844026+0000", 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015368900000000001 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844058+0000", 2024-09-21T23:26:37.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1449999999999999e-05 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844077+0000", 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9310999999999999e-05 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 75.542707970999999, 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068324850000000001, 2024-09-21T23:26:37.513 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 141 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:37.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:37.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.22e-07 2024-09-21T23:26:37.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837539+0000", 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.023e-06 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837543+0000", 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6490000000000001e-06 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837546+0000", 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.382e-06 2024-09-21T23:26:37.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843479+0000", 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059328189999999998 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843519+0000", 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0071000000000003e-05 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843600+0000", 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1376000000000002e-05 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844137+0000", 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053640299999999997 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844319+0000", 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000181893 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844346+0000", 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.781e-05 2024-09-21T23:26:37.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844366+0000", 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9737000000000001e-05 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 74.927374275999995, 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006594145, 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.519 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 158 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452868+0000", 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9500000000000003e-07 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452938+0000", 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9598999999999996e-05 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452950+0000", 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2971e-05 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452960+0000", 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.448e-06 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456217+0000", 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003257378 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456557+0000", 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033989399999999998 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456703+0000", 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000145599 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459339+0000", 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026365529999999998 2024-09-21T23:26:37.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459386+0000", 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6640000000000001e-05 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459428+0000", 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2148999999999999e-05 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459462+0000", 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3419e-05 2024-09-21T23:26:37.524 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 74.927269902000006, 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0065672760000000004, 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:37.525 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 159 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:37.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5400000000000001e-07 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452977+0000", 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.301e-06 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452984+0000", 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1840000000000002e-06 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452989+0000", 2024-09-21T23:26:37.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5219999999999998e-06 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457324+0000", 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043348780000000003 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457416+0000", 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1824999999999996e-05 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457574+0000", 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015810999999999999 2024-09-21T23:26:37.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459376+0000", 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001801649 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459467+0000", 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1367999999999997e-05 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459507+0000", 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9576000000000002e-05 2024-09-21T23:26:37.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:37.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:37.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:37.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459539+0000", 2024-09-21T23:26:37.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2308999999999998e-05 2024-09-21T23:26:37.530 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.530 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.530 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.530 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:37.530 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:37.530 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:37.531 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:37.531+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:37.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:37.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:26:37.632 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:37.631+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:37.643 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:37.643 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:37.643 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:37.643 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:37.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:37.704 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:37.704 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:37.723 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:37.733 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:37.732+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:37.742 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:37.759 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:37.759 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:37.777 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:37.777 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:37.795 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:37.795 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:37.807 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:37.807 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:37.812 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:37.813 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:37.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:37.830 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:37.830 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:37.834 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:37.833+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:37.935 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:37.934+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:37.939 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:37.939 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:37.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:38.036 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:38.036+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:38.096 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:38.096 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:38.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:38.137 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:38.136+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:38.238 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:38.237+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:38.253 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:38.253 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:38.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:38.339 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:38.338+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:38.414 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:38.414 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:38.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:38.467 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:38.467+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:38.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T23:26:38.536 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:38.536 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:38.540 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:38.539+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:38.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:38.641 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:38.640+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:38.700 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:38.701 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:38.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:38.753 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:38.753+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:38.828 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:38.828 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:38.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:38.878 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:38.878+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:38.944 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:38.943+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:38.986 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:38.986 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:38.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:39.045 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:39.044+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:39.104 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:39.104 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:39.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:39.146 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:39.145+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:39.247 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:39.246+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:39.261 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:39.261 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:39.271 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:39.348 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:39.348+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:39.391 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:39.391 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:39.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:39.448 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:39.447+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:39.490 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-09-21T23:26:39.508 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:39.508 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:39.517 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:39.558 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:39.557+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:39.650 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:39.649+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:39.664 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:39.665 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:39.674 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:39.751 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:39.751+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:39.823 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:39.823 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:39.832 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:39.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:26:39.876 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:39.875+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:39.884 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 93% (12018/12896) Updating files: 94% (12123/12896) Updating files: 95% (12252/12896) Updating files: 96% (12381/12896) Updating files: 97% (12510/12896) Updating files: 98% (12639/12896) Updating files: 99% (12768/12896) Updating files: 100% (12896/12896) Updating files: 100% (12896/12896), done. 2024-09-21T23:26:39.939 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:39.939 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:39.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:39.953 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:39.953+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:39.973 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:39.973 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:39.990 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:39.990 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:40.006 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:40.006 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:40.022 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:40.023 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:40.039 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:40.039 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:40.054 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:40.053+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:40.055 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:40.056 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:40.072 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:40.072 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:40.089 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:40.089 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:40.096 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:40.096 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:40.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:40.096 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 71.932527297000007, 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0080986310000000002, 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.097 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163709+0000", 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163727+0000", 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7552000000000001e-05 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163744+0000", 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7541999999999999e-05 2024-09-21T23:26:40.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163757+0000", 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2855e-05 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164069+0000", 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031179899999999997 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164084+0000", 2024-09-21T23:26:40.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5265e-05 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167127+0000", 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030427029999999999 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167317+0000", 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000190006 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167465+0000", 2024-09-21T23:26:40.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014824 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.168309+0000", 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084350899999999999 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171730+0000", 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034217620000000001 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171770+0000", 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0074999999999998e-05 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171802+0000", 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1563999999999998e-05 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.104 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:1 2.3 e19/18 ondisk, result = 0)", 2024-09-21T23:26:40.104 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:40.104 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 71.924585050000005, 2024-09-21T23:26:40.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016281800000000001, 2024-09-21T23:26:40.104 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:40.104 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:40.104 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:40.104 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:40.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:40.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:40.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:40.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:40.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171646+0000", 2024-09-21T23:26:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6899999999999996e-07 2024-09-21T23:26:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171651+0000", 2024-09-21T23:26:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0309999999999998e-06 2024-09-21T23:26:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171665+0000", 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.309e-05 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171672+0000", 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6070000000000002e-06 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171716+0000", 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4344000000000001e-05 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171727+0000", 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0488e-05 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171808+0000", 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1489000000000005e-05 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.110 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 71.921809608000004, 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023803299999999999, 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 2 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:40.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174421+0000", 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174422+0000", 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4300000000000001e-06 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174429+0000", 2024-09-21T23:26:40.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1659999999999999e-06 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174443+0000", 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4358e-05 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174449+0000", 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2770000000000002e-06 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174540+0000", 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0209000000000003e-05 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174659+0000", 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011959300000000001 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:40.114 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 71.921344607999998, 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016795100000000001, 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 3 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:40.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174886+0000", 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174887+0000", 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:40.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174892+0000", 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8180000000000003e-06 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174902+0000", 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0095000000000001e-05 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174911+0000", 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0799999999999995e-06 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.174974+0000", 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3027000000000005e-05 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.175054+0000", 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0011000000000004e-05 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T23:26:40.118 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 70.927838976999993, 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006147263, 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:40.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168394+0000", 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.956e-06 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168404+0000", 2024-09-21T23:26:40.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0463e-05 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168423+0000", 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8715999999999998e-05 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168432+0000", 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.516e-06 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168529+0000", 2024-09-21T23:26:40.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7073999999999999e-05 2024-09-21T23:26:40.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:40.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168534+0000", 2024-09-21T23:26:40.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.831e-06 2024-09-21T23:26:40.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170224+0000", 2024-09-21T23:26:40.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016893590000000001 2024-09-21T23:26:40.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:40.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170332+0000", 2024-09-21T23:26:40.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108346 2024-09-21T23:26:40.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:40.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170445+0000", 2024-09-21T23:26:40.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112642 2024-09-21T23:26:40.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:40.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.171210+0000", 2024-09-21T23:26:40.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076543800000000001 2024-09-21T23:26:40.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174469+0000", 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003259086 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174506+0000", 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6465999999999999e-05 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174539+0000", 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3370000000000001e-05 2024-09-21T23:26:40.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:5 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 70.921896931999996, 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021298500000000001, 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:40.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174334+0000", 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174335+0000", 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.111e-06 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:40.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174341+0000", 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6239999999999996e-06 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174356+0000", 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4548e-05 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174375+0000", 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8757e-05 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174457+0000", 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2702999999999997e-05 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174466+0000", 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.208e-06 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174547+0000", 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1033999999999997e-05 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.131 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 69.912124892999998, 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040438610000000002, 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:40.132 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184106+0000", 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:40.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184109+0000", 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9069999999999999e-06 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184124+0000", 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5780000000000001e-05 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184159+0000", 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4106999999999997e-05 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184178+0000", 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9908999999999999e-05 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184330+0000", 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015152300000000001 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.184337+0000", 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7920000000000004e-06 2024-09-21T23:26:40.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186239+0000", 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019020129999999999 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186307+0000", 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8326999999999998e-05 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:40.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.186459+0000", 2024-09-21T23:26:40.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152099 2024-09-21T23:26:40.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:40.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187261+0000", 2024-09-21T23:26:40.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00080186199999999997 2024-09-21T23:26:40.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:40.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188085+0000", 2024-09-21T23:26:40.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00082417199999999997 2024-09-21T23:26:40.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:40.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188120+0000", 2024-09-21T23:26:40.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5194999999999998e-05 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188150+0000", 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9175000000000001e-05 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.4244.0:7 2.2 e20/18 ondisk, result = 0)", 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 69.908254157000002, 2024-09-21T23:26:40.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000179505, 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187977+0000", 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.16e-07 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.187985+0000", 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3039999999999999e-06 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188000+0000", 2024-09-21T23:26:40.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5097e-05 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188007+0000", 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1330000000000001e-06 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188074+0000", 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7514000000000004e-05 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188082+0000", 2024-09-21T23:26:40.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3649999999999998e-06 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:30.188156+0000", 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4176000000000003e-05 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4224.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 26.300777435000001, 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022720800000000001, 2024-09-21T23:26:40.145 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4224", 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/3116157232", 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:40.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:40.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795453+0000", 2024-09-21T23:26:40.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:40.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:40.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795455+0000", 2024-09-21T23:26:40.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0609999999999998e-06 2024-09-21T23:26:40.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:40.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795473+0000", 2024-09-21T23:26:40.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7762e-05 2024-09-21T23:26:40.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:40.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795487+0000", 2024-09-21T23:26:40.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.412e-05 2024-09-21T23:26:40.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:40.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795496+0000", 2024-09-21T23:26:40.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8780000000000004e-06 2024-09-21T23:26:40.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:40.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795578+0000", 2024-09-21T23:26:40.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1700999999999996e-05 2024-09-21T23:26:40.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:40.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:40.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:40.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:13.795680+0000", 2024-09-21T23:26:40.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102686 2024-09-21T23:26:40.152 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.152 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:40.152 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.152 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:40.152 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:40.152 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:40.154 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:40.195 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:40.194+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:40.255 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:40.255+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:40.301 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:40.301 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:40.301 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:40.301 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:40.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:40.357 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:40.356+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:40.417 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:40.418 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:40.418 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:40.418 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:40.418 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:40.426 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:40.467 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:40.466+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:40.558 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:40.557+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:40.573 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:40.573 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:40.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:40.573 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-09-21T23:26:40.573 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:40.582 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:40.659 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:40.658+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:40.729 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:40.729 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:40.729 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:40.730 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:40.738 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:40.782 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:40.781+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:40.838 INFO:tasks.workunit.client.0.smithi031.stderr:Note: switching to '4e45e6b490e6cbe082cdf155fea780438ec53fc7'. 2024-09-21T23:26:40.838 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-21T23:26:40.838 INFO:tasks.workunit.client.0.smithi031.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-21T23:26:40.838 INFO:tasks.workunit.client.0.smithi031.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-21T23:26:40.838 INFO:tasks.workunit.client.0.smithi031.stderr:state without impacting any branches by switching back to a branch. 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr: git switch -c 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr:Or undo this operation with: 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr: git switch - 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-21T23:26:40.839 INFO:tasks.workunit.client.0.smithi031.stderr:HEAD is now at 4e45e6b490e Merge pull request #59907 from zdover23/wip-doc-2024-09-21-backport-59904-to-squid 2024-09-21T23:26:40.844 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T23:26:40.845 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:40.845 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:40.845 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:40.845 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:40.845 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:40.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:40.894 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:40.893+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:40.908 INFO:tasks.workunit.client.0.smithi031.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T23:26:40.910 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T23:26:40.910 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T23:26:40.961 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:40.961+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:41.002 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:41.002 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:41.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:41.002 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:41.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.002 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.002 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:41.002 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.833372065000006, 2024-09-21T23:26:41.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090066050000000009, 2024-09-21T23:26:41.002 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 37 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7099999999999998e-07 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167009+0000", 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.92e-06 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167013+0000", 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167017+0000", 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7280000000000001e-06 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170778+0000", 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037612399999999999 2024-09-21T23:26:41.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170831+0000", 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2694000000000001e-05 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170970+0000", 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000138966 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.171555+0000", 2024-09-21T23:26:41.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058538399999999999 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175872+0000", 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043172389999999996 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175969+0000", 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6844999999999999e-05 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.176012+0000", 2024-09-21T23:26:41.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3356999999999998e-05 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.807267229000004, 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069023069999999999, 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.008 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0999999999999999e-07 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193115+0000", 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5700000000000001e-06 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193129+0000", 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4124000000000001e-05 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193133+0000", 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0550000000000001e-06 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194144+0000", 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001010735 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194231+0000", 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7665999999999997e-05 2024-09-21T23:26:41.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194357+0000", 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125353 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194844+0000", 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000487305 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199964+0000", 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051204859999999996 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199992+0000", 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7902000000000002e-05 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.200013+0000", 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0601000000000001e-05 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.785618154999995, 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0067494620000000003, 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 73 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4900000000000001e-07 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214818+0000", 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7754000000000003e-05 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214842+0000", 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4552000000000001e-05 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214846+0000", 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9129999999999996e-06 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215813+0000", 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00096695899999999998 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215904+0000", 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0328000000000007e-05 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216020+0000", 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115838 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217018+0000", 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099836200000000003 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221461+0000", 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044435969999999997 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221487+0000", 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5806e-05 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221509+0000", 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1903999999999999e-05 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.019 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.785523452000007, 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068139230000000004, 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 74 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:41.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214855+0000", 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-07 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214859+0000", 2024-09-21T23:26:41.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1389999999999997e-06 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214871+0000", 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1698999999999999e-05 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214887+0000", 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6061000000000001e-05 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216359+0000", 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014720779999999999 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216439+0000", 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0191999999999994e-05 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216579+0000", 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014029099999999999 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217129+0000", 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054932099999999997 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221559+0000", 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044299710000000004 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221627+0000", 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8631000000000001e-05 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221668+0000", 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0988000000000001e-05 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.768847265999995, 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0077112509999999997, 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.025 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 86 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5199999999999998e-07 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231533+0000", 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3879999999999998e-06 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231537+0000", 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0010000000000003e-06 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231540+0000", 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0489999999999999e-06 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233890+0000", 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023500249999999999 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233937+0000", 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6400999999999998e-05 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234015+0000", 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8343000000000005e-05 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235675+0000", 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001660124 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239138+0000", 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034624109999999999 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239200+0000", 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2698000000000002e-05 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239242+0000", 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1458999999999997e-05 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.030 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.768830996999995, 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010200097999999999, 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.031 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2700000000000001e-07 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231588+0000", 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0985000000000001e-05 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231593+0000", 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1059999999999999e-06 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231606+0000", 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2343e-05 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234406+0000", 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027999850000000001 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234489+0000", 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3051000000000002e-05 2024-09-21T23:26:41.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234631+0000", 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285799999999999 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235679+0000", 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001047517 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241644+0000", 2024-09-21T23:26:41.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059652129999999996 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241704+0000", 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9732999999999999e-05 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241747+0000", 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-05 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.768765157999994, 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010246534, 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 88 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.037 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0199999999999998e-07 2024-09-21T23:26:41.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231615+0000", 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3439999999999999e-06 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231619+0000", 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0219999999999998e-06 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231622+0000", 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6989999999999999e-06 2024-09-21T23:26:41.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235337+0000", 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037150030000000001 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235385+0000", 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7861999999999997e-05 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235458+0000", 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2596999999999996e-05 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235978+0000", 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052013600000000004 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241799+0000", 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058215289999999998 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241836+0000", 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6282999999999998e-05 2024-09-21T23:26:41.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241859+0000", 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3757e-05 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.611373975000006, 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072206969999999999, 2024-09-21T23:26:41.042 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 112 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.72e-07 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389069+0000", 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4875000000000005e-05 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389079+0000", 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0135e-05 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389085+0000", 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394006+0000", 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049207920000000002 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394052+0000", 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5810999999999999e-05 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394139+0000", 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7130000000000001e-05 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395407+0000", 2024-09-21T23:26:41.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012681260000000001 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396139+0000", 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000731703 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396194+0000", 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5309999999999997e-05 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396225+0000", 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0589000000000001e-05 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.611267018000007, 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086923490000000003, 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.048 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 113 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:41.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389116+0000", 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.245e-06 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389126+0000", 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0295000000000001e-05 2024-09-21T23:26:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389132+0000", 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0889999999999996e-06 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394548+0000", 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005415876 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394642+0000", 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4531999999999995e-05 2024-09-21T23:26:41.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394714+0000", 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1753000000000002e-05 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396233+0000", 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015185299999999999 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397749+0000", 2024-09-21T23:26:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015157770000000001 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397779+0000", 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0561000000000001e-05 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397803+0000", 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4116000000000001e-05 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.053 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.611234948999993, 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.01139824, 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 114 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.054 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.01e-07 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389146+0000", 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5239999999999999e-06 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389150+0000", 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3989999999999997e-06 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389153+0000", 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.038e-06 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395026+0000", 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058731720000000003 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395071+0000", 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4388999999999998e-05 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395141+0000", 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0053000000000002e-05 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397196+0000", 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020555899999999999 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400494+0000", 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032972679999999999 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400520+0000", 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6423000000000001e-05 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400541+0000", 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0982999999999999e-05 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.61121627, 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011537824, 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.059 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 115 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4900000000000001e-07 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389197+0000", 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5204999999999999e-05 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389203+0000", 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0979999999999999e-06 2024-09-21T23:26:41.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389207+0000", 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6600000000000001e-06 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395443+0000", 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062358099999999996 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395525+0000", 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1855999999999997e-05 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395669+0000", 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014472399999999999 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397204+0000", 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001534817 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400599+0000", 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033944499999999998 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400655+0000", 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6647000000000002e-05 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400699+0000", 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4208000000000001e-05 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.064 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.611164818000006, 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011571580999999999, 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 116 2024-09-21T23:26:41.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3099999999999999e-07 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389216+0000", 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4260000000000002e-06 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389223+0000", 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389225+0000", 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7760000000000002e-06 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396256+0000", 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070309020000000003 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396348+0000", 2024-09-21T23:26:41.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1780000000000006e-05 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396481+0000", 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000133101 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397208+0000", 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072716199999999995 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400737+0000", 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035290220000000001 2024-09-21T23:26:41.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400763+0000", 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5219999999999999e-05 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400785+0000", 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2045e-05 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.611148134999993, 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011687858000000001, 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 117 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.071 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3699999999999999e-07 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389232+0000", 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1789999999999998e-06 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389238+0000", 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209999999999999e-06 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389241+0000", 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.557e-06 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396792+0000", 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075506899999999997 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396838+0000", 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5639000000000001e-05 2024-09-21T23:26:41.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396958+0000", 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119892 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397458+0000", 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049987600000000001 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400824+0000", 2024-09-21T23:26:41.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033662560000000002 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400880+0000", 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5779999999999998e-05 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400918+0000", 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8031000000000002e-05 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.599129661000006, 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006886924, 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 120 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401249+0000", 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.13e-07 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401286+0000", 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6773999999999997e-05 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401295+0000", 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9039999999999998e-06 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401299+0000", 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3250000000000001e-06 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402183+0000", 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000883758 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402264+0000", 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1080000000000003e-05 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402401+0000", 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013719700000000001 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403430+0000", 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010290119999999999 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408045+0000", 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046146659999999999 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408095+0000", 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0898e-05 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408135+0000", 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9697000000000003e-05 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.599071187000007, 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069210920000000002, 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 121 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3200000000000001e-07 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401341+0000", 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4236999999999998e-05 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401346+0000", 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.04e-06 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401350+0000", 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6399999999999999e-06 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402733+0000", 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013825790000000001 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402816+0000", 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3461000000000006e-05 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402941+0000", 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012518300000000001 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403533+0000", 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000592071 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408174+0000", 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046404209999999996 2024-09-21T23:26:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408204+0000", 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0147000000000001e-05 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408228+0000", 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3981e-05 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.162928289999996, 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064924400000000004, 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 139 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0800000000000001e-07 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837452+0000", 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7209999999999999e-06 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837455+0000", 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2279999999999998e-06 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837458+0000", 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8200000000000001e-06 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842187+0000", 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047298460000000002 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842233+0000", 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6056000000000001e-05 2024-09-21T23:26:41.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842304+0000", 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0862e-05 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843291+0000", 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00098632100000000007 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843894+0000", 2024-09-21T23:26:41.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060369199999999997 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843922+0000", 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7429000000000001e-05 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843942+0000", 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0256999999999999e-05 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.162916347000007, 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066155550000000004, 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 140 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.094 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:41.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1e-07 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837517+0000", 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5034000000000002e-05 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837522+0000", 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8520000000000003e-06 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837525+0000", 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4139999999999998e-06 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842562+0000", 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050366279999999996 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842615+0000", 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3675999999999999e-05 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842715+0000", 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9990999999999999e-05 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843873+0000", 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011573 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844026+0000", 2024-09-21T23:26:41.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015368900000000001 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844058+0000", 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1449999999999999e-05 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844077+0000", 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9310999999999999e-05 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.162844153999998, 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068324850000000001, 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 141 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.100 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.22e-07 2024-09-21T23:26:41.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837539+0000", 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.023e-06 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837543+0000", 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6490000000000001e-06 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837546+0000", 2024-09-21T23:26:41.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.382e-06 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843479+0000", 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059328189999999998 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843519+0000", 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0071000000000003e-05 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843600+0000", 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1376000000000002e-05 2024-09-21T23:26:41.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844137+0000", 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053640299999999997 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844319+0000", 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000181893 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844346+0000", 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.781e-05 2024-09-21T23:26:41.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844366+0000", 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9737000000000001e-05 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.547510458999994, 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006594145, 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.105 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 158 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452868+0000", 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9500000000000003e-07 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452938+0000", 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9598999999999996e-05 2024-09-21T23:26:41.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452950+0000", 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2971e-05 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452960+0000", 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.448e-06 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456217+0000", 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003257378 2024-09-21T23:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456557+0000", 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033989399999999998 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456703+0000", 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000145599 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459339+0000", 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026365529999999998 2024-09-21T23:26:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459386+0000", 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6640000000000001e-05 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459428+0000", 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2148999999999999e-05 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459462+0000", 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3419e-05 2024-09-21T23:26:41.110 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.547406085000006, 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0065672760000000004, 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:41.111 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 159 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:41.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5400000000000001e-07 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452977+0000", 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.301e-06 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452984+0000", 2024-09-21T23:26:41.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1840000000000002e-06 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452989+0000", 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5219999999999998e-06 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457324+0000", 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043348780000000003 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457416+0000", 2024-09-21T23:26:41.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1824999999999996e-05 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457574+0000", 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015810999999999999 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459376+0000", 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001801649 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:41.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459467+0000", 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1367999999999997e-05 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459507+0000", 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9576000000000002e-05 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459539+0000", 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2308999999999998e-05 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.116 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.117 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:41.117 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:41.117 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:41.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:41.119 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:41.118+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:41.219 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:41.219+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:41.226 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:41.226 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:41.226 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:41.226 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:41.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:41.319 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:41.319+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:41.382 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:41.382 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:41.382 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:41.382 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:41.382 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:41.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:41.431 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:41.431+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:41.504 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:26:41.504 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:26:41.504 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T23:26:41.504 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-09-21T23:26:41.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-21T23:26:41.505 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T23:26:41.543 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:41.543 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:41.543 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:41.543 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.404687781000007, 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036127939999999999, 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3500000000000002e-07 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136608+0000", 2024-09-21T23:26:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.182e-06 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136618+0000", 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9199999999999999e-06 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136628+0000", 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4390000000000006e-06 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.546 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136692+0000", 2024-09-21T23:26:41.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3979000000000005e-05 2024-09-21T23:26:41.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136716+0000", 2024-09-21T23:26:41.547 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4391000000000001e-05 2024-09-21T23:26:41.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.547 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.547 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140171+0000", 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034554910000000002 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140214+0000", 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2057000000000001e-05 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'18)", 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.370169618000006, 2024-09-21T23:26:41.548 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043957730000000004, 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171120+0000", 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6700000000000003e-07 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171125+0000", 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.451e-06 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171138+0000", 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2541000000000001e-05 2024-09-21T23:26:41.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171143+0000", 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4659999999999998e-06 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171187+0000", 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4014000000000002e-05 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171200+0000", 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3108999999999999e-05 2024-09-21T23:26:41.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175479+0000", 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042784570000000003 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175515+0000", 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5967999999999999e-05 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:58 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-21T23:26:41.553 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:41.553 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.347417321999998, 2024-09-21T23:26:41.553 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003032851, 2024-09-21T23:26:41.553 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.553 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.553 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.553 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193872+0000", 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9400000000000004e-07 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193952+0000", 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9791999999999998e-05 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193962+0000", 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0034000000000001e-05 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.555 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193973+0000", 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0733e-05 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194015+0000", 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2911999999999998e-05 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194031+0000", 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5082999999999999e-05 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196869+0000", 2024-09-21T23:26:41.556 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002838808 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196904+0000", 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4694999999999999e-05 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:59 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-21T23:26:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.346831656000006, 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053675709999999998, 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:41.558 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7100000000000002e-07 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194462+0000", 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.668e-06 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.559 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194469+0000", 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1670000000000002e-06 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194475+0000", 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-06 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194513+0000", 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7391999999999997e-05 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.560 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194531+0000", 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7870999999999999e-05 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199791+0000", 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052604119999999999 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199824+0000", 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3340000000000003e-05 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.561 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:60 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'38, mlcod=16'36)", 2024-09-21T23:26:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.340750960999998, 2024-09-21T23:26:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003614562, 2024-09-21T23:26:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200539+0000", 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5799999999999999e-06 2024-09-21T23:26:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200551+0000", 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1816e-05 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200578+0000", 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6772999999999999e-05 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200590+0000", 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2303000000000001e-05 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200715+0000", 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012523499999999999 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200731+0000", 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5690000000000001e-05 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204121+0000", 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003390363 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204152+0000", 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0802000000000001e-05 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:70 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'39)", 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.330695305999996, 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033896579999999998, 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210594+0000", 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1099999999999995e-07 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210600+0000", 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4389999999999997e-06 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210613+0000", 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.243e-05 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.568 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210618+0000", 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0939999999999998e-06 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210843+0000", 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022493200000000001 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210858+0000", 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5109000000000001e-05 2024-09-21T23:26:41.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213938+0000", 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030796500000000002 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213983+0000", 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5293000000000002e-05 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:72 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'46, mlcod=16'43)", 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.325750529999993, 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029102279999999999, 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215539+0000", 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4199999999999995e-07 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215589+0000", 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0488999999999997e-05 2024-09-21T23:26:41.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215598+0000", 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2830000000000007e-06 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215610+0000", 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1134999999999999e-05 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215657+0000", 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7658999999999999e-05 2024-09-21T23:26:41.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215679+0000", 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1526999999999999e-05 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218410+0000", 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027316939999999998 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218448+0000", 2024-09-21T23:26:41.574 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7799000000000002e-05 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:73 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'47, mlcod=16'43)", 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.325141183, 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051882020000000003, 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216148+0000", 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7100000000000001e-07 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.576 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216218+0000", 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0313000000000003e-05 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216227+0000", 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9959999999999999e-06 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216232+0000", 2024-09-21T23:26:41.577 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5340000000000001e-06 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216267+0000", 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4739000000000001e-05 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216281+0000", 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4345000000000001e-05 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221290+0000", 2024-09-21T23:26:41.578 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050088989999999998 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221336+0000", 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5704999999999997e-05 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:74 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'48, mlcod=16'43)", 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.32461524, 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046922939999999996, 2024-09-21T23:26:41.579 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216674+0000", 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9900000000000001e-07 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216679+0000", 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.502e-06 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216687+0000", 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1790000000000008e-06 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216691+0000", 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7230000000000002e-06 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216737+0000", 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6087000000000001e-05 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216758+0000", 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0434000000000001e-05 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221338+0000", 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045804569999999996 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221366+0000", 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7413e-05 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:86 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'56, mlcod=16'50)", 2024-09-21T23:26:41.583 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.307193306000002, 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049055269999999998, 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234096+0000", 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234099+0000", 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3890000000000001e-06 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234107+0000", 2024-09-21T23:26:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2349999999999999e-06 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234112+0000", 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0629999999999999e-06 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234143+0000", 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1131999999999999e-05 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234155+0000", 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2140000000000001e-05 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.238972+0000", 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048167770000000004 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.239001+0000", 2024-09-21T23:26:41.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9164000000000001e-05 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:87 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'57, mlcod=16'52)", 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.306513930999998, 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067179910000000004, 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0299999999999998e-07 2024-09-21T23:26:41.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234813+0000", 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7531999999999999e-05 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234826+0000", 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2617e-05 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234831+0000", 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1229999999999999e-06 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234869+0000", 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8513999999999998e-05 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234881+0000", 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2148e-05 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241465+0000", 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065837329999999996 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241493+0000", 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7521e-05 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:88 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:41.592 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.305716082999993, 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059344530000000001, 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235573+0000", 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0100000000000004e-07 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235578+0000", 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1590000000000001e-06 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.594 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235590+0000", 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1229e-05 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235595+0000", 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0889999999999999e-06 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235943+0000", 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000348227 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235957+0000", 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4419999999999999e-05 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241494+0000", 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055371070000000003 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241507+0000", 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2520999999999999e-05 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.596 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:113 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'75, mlcod=16'71)", 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.146484805, 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028543489999999999, 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8199999999999999e-07 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394809+0000", 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-09-21T23:26:41.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394821+0000", 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2024999999999999e-05 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394827+0000", 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0020000000000003e-06 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394862+0000", 2024-09-21T23:26:41.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4811000000000003e-05 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394873+0000", 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.058e-05 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397631+0000", 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027581060000000002 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397658+0000", 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7073999999999999e-05 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'71)", 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.146073392000005, 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005160085, 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395216+0000", 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4999999999999998e-07 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.602 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395221+0000", 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2290000000000002e-06 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395229+0000", 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2369999999999992e-06 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395233+0000", 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1840000000000001e-06 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395260+0000", 2024-09-21T23:26:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.728e-05 2024-09-21T23:26:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395272+0000", 2024-09-21T23:26:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1270000000000001e-05 2024-09-21T23:26:41.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400341+0000", 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050691809999999999 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400375+0000", 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4254000000000001e-05 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'72)", 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.145524871000006, 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046274580000000001, 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395829+0000", 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4618999999999999e-05 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395836+0000", 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9090000000000003e-06 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395839+0000", 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.371e-06 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395868+0000", 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9328000000000001e-05 2024-09-21T23:26:41.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395880+0000", 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1501000000000001e-05 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400376+0000", 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044966340000000002 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400391+0000", 2024-09-21T23:26:41.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4676e-05 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:116 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'78, mlcod=16'73)", 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.144720324000005, 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038400970000000002, 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396569+0000", 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1100000000000001e-07 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396573+0000", 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5569999999999997e-06 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396579+0000", 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1120000000000001e-06 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396587+0000", 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396630+0000", 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2991000000000002e-05 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396642+0000", 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1481000000000001e-05 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400392+0000", 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037507830000000002 2024-09-21T23:26:41.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400408+0000", 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5863e-05 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:117 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'79, mlcod=16'73)", 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.144217982000001, 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033512120000000001, 2024-09-21T23:26:41.615 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7e-07 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397075+0000", 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7809999999999999e-06 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397082+0000", 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-09-21T23:26:41.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397085+0000", 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7139999999999999e-06 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397130+0000", 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.498e-05 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397142+0000", 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1838000000000001e-05 2024-09-21T23:26:41.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400409+0000", 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032666349999999999 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400422+0000", 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2866e-05 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.139394308999996, 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034761240000000001, 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401895+0000", 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7400000000000003e-07 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401963+0000", 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8701000000000005e-05 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401972+0000", 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9530000000000005e-06 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401977+0000", 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1579999999999998e-06 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402009+0000", 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2512000000000003e-05 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402022+0000", 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3220999999999999e-05 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405337+0000", 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033150300000000001 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405370+0000", 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2975000000000002e-05 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.138788238999993, 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054168369999999999, 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:41.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402501+0000", 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2699999999999999e-07 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402589+0000", 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899000000000007e-05 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402599+0000", 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0312000000000001e-05 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402605+0000", 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9689999999999999e-06 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402648+0000", 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3210000000000001e-05 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402664+0000", 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5347999999999999e-05 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407887+0000", 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052237919999999997 2024-09-21T23:26:41.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407917+0000", 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9779999999999999e-05 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 80.138258858, 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049049180000000003, 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:41.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4099999999999999e-07 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403098+0000", 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7464000000000003e-05 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403106+0000", 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6629999999999999e-06 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:41.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403111+0000", 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403137+0000", 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6349000000000001e-05 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403148+0000", 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.077e-05 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:41.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407920+0000", 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047716590000000001 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407935+0000", 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4783999999999999e-05 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:41.632 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:41.633 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:41.542+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:41.633 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T23:26:41.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:41.634 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T23:26:41.666 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T23:26:41.666 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:41.665+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:41.668 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T23:26:41.668 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T23:26:41.717 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T23:26:41.721 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:26:41.721 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T23:26:41.735 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:41.735+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:41.737 INFO:tasks.workunit:Running workunits matching rbd/journal.sh on client.0... 2024-09-21T23:26:41.738 INFO:tasks.workunit:Running workunit rbd/journal.sh... 2024-09-21T23:26:41.738 DEBUG:teuthology.orchestra.run.smithi031:workunit test rbd/journal.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e45e6b490e6cbe082cdf155fea780438ec53fc7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/journal.sh 2024-09-21T23:26:41.776 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:41.777 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:41.777 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:41.777 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:41.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:41.800 INFO:tasks.workunit.client.0.smithi031.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-21T23:26:41.806 INFO:tasks.workunit.client.0.smithi031.stderr:+ test_rbd_journal 2024-09-21T23:26:41.806 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdjournal46970 2024-09-21T23:26:41.806 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --image-feature exclusive-lock --image-feature journaling --size 128 testrbdjournal46970 2024-09-21T23:26:41.837 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:41.836+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:41.867 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd info testrbdjournal46970 --format=xml 2024-09-21T23:26:41.867 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //image/journal 2024-09-21T23:26:41.874 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T23:26:41.874 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-09-21T23:26:41.916 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:41.916 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:41.916 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:41.916 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:41.916 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:41.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:41.930 INFO:tasks.workunit.client.0.smithi031.stderr:+ local journal=1286a02efebc 2024-09-21T23:26:41.930 INFO:tasks.workunit.client.0.smithi031.stderr:+ test -n 1286a02efebc 2024-09-21T23:26:41.931 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal info 1286a02efebc 2024-09-21T23:26:41.937 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:41.937+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:41.967 INFO:tasks.workunit.client.0.smithi031.stdout:rbd journal '1286a02efebc': 2024-09-21T23:26:41.967 INFO:tasks.workunit.client.0.smithi031.stdout: header_oid: journal.1286a02efebc 2024-09-21T23:26:41.968 INFO:tasks.workunit.client.0.smithi031.stdout: object_oid_prefix: journal_data.2.1286a02efebc. 2024-09-21T23:26:41.968 INFO:tasks.workunit.client.0.smithi031.stdout: order: 24 (16 MiB objects) 2024-09-21T23:26:41.968 INFO:tasks.workunit.client.0.smithi031.stdout: splay_width: 4 2024-09-21T23:26:41.971 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal info --journal 1286a02efebc 2024-09-21T23:26:42.007 INFO:tasks.workunit.client.0.smithi031.stdout:rbd journal '1286a02efebc': 2024-09-21T23:26:42.007 INFO:tasks.workunit.client.0.smithi031.stdout: header_oid: journal.1286a02efebc 2024-09-21T23:26:42.007 INFO:tasks.workunit.client.0.smithi031.stdout: object_oid_prefix: journal_data.2.1286a02efebc. 2024-09-21T23:26:42.007 INFO:tasks.workunit.client.0.smithi031.stdout: order: 24 (16 MiB objects) 2024-09-21T23:26:42.007 INFO:tasks.workunit.client.0.smithi031.stdout: splay_width: 4 2024-09-21T23:26:42.011 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal info --image testrbdjournal46970 2024-09-21T23:26:42.038 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:42.037+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:42.046 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:42.046 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:42.046 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:42.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.046 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-21T23:26:42.046 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:42.046 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 72.875357464000004, 2024-09-21T23:26:42.046 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003527526, 2024-09-21T23:26:42.046 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:42.046 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:42.047 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170743+0000", 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.08e-07 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170750+0000", 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800000000000001e-06 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170767+0000", 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6923999999999999e-05 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170788+0000", 2024-09-21T23:26:42.048 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0446e-05 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170849+0000", 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1409999999999996e-05 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170881+0000", 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2075e-05 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174218+0000", 2024-09-21T23:26:42.049 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033371070000000002 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174270+0000", 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1375999999999997e-05 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 71.859367969999994, 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0011783570000000001, 2024-09-21T23:26:42.050 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:42.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186732+0000", 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7999999999999996e-07 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186750+0000", 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7444000000000001e-05 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186834+0000", 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3974999999999995e-05 2024-09-21T23:26:42.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186876+0000", 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2796000000000002e-05 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.186999+0000", 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012303800000000001 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187036+0000", 2024-09-21T23:26:42.053 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6897000000000003e-05 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187869+0000", 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00083215199999999996 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:30.187910+0000", 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1575000000000001e-05 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:42.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:42.055 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:42.055 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:42.056 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:42.062 INFO:tasks.workunit.client.0.smithi031.stdout:rbd journal '1286a02efebc': 2024-09-21T23:26:42.062 INFO:tasks.workunit.client.0.smithi031.stdout: header_oid: journal.1286a02efebc 2024-09-21T23:26:42.062 INFO:tasks.workunit.client.0.smithi031.stdout: object_oid_prefix: journal_data.2.1286a02efebc. 2024-09-21T23:26:42.062 INFO:tasks.workunit.client.0.smithi031.stdout: order: 24 (16 MiB objects) 2024-09-21T23:26:42.062 INFO:tasks.workunit.client.0.smithi031.stdout: splay_width: 4 2024-09-21T23:26:42.066 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd feature disable testrbdjournal46970 journaling 2024-09-21T23:26:42.130 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd info testrbdjournal46970 --format=xml 2024-09-21T23:26:42.131 INFO:tasks.workunit.client.0.smithi031.stderr:+ expect_false xmlstarlet sel -t -v //image/journal 2024-09-21T23:26:42.131 INFO:tasks.workunit.client.0.smithi031.stderr:+ set -x 2024-09-21T23:26:42.131 INFO:tasks.workunit.client.0.smithi031.stderr:+ xmlstarlet sel -t -v //image/journal 2024-09-21T23:26:42.139 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:42.138+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:42.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:26:42.177 INFO:tasks.workunit.client.0.smithi031.stderr:+ return 0 2024-09-21T23:26:42.177 INFO:tasks.workunit.client.0.smithi031.stderr:+ expect_false rbd journal info 1286a02efebc 2024-09-21T23:26:42.177 INFO:tasks.workunit.client.0.smithi031.stderr:+ set -x 2024-09-21T23:26:42.178 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal info 1286a02efebc 2024-09-21T23:26:42.181 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:42.181 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:42.181 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:42.181 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:42.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:42.214 INFO:tasks.workunit.client.0.smithi031.stderr:failed to get journal metadata: (2) No such file or directory 2024-09-21T23:26:42.214 INFO:tasks.workunit.client.0.smithi031.stderr:rbd: journal info: (2) No such file or directory 2024-09-21T23:26:42.217 INFO:tasks.workunit.client.0.smithi031.stderr:+ return 0 2024-09-21T23:26:42.217 INFO:tasks.workunit.client.0.smithi031.stderr:+ expect_false rbd journal info --image testrbdjournal46970 2024-09-21T23:26:42.217 INFO:tasks.workunit.client.0.smithi031.stderr:+ set -x 2024-09-21T23:26:42.217 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal info --image testrbdjournal46970 2024-09-21T23:26:42.239 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:42.238+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:42.261 INFO:tasks.workunit.client.0.smithi031.stderr:rbd: journaling is not enabled for image testrbdjournal46970 2024-09-21T23:26:42.265 INFO:tasks.workunit.client.0.smithi031.stderr:+ return 0 2024-09-21T23:26:42.265 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd feature enable testrbdjournal46970 journaling 2024-09-21T23:26:42.305 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:42.325 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:42.326 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:42.328 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd info testrbdjournal46970 --format=xml 2024-09-21T23:26:42.328 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //image/journal 2024-09-21T23:26:42.340 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:42.340+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:42.358 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:42.359 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:42.359 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:42.359 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:42.359 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:42.360 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:42.360 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:42.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:42.377 INFO:tasks.workunit.client.0.smithi031.stderr:+ local journal1=1286a02efebc 2024-09-21T23:26:42.377 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 1286a02efebc = 1286a02efebc 2024-09-21T23:26:42.377 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal info 1286a02efebc 2024-09-21T23:26:42.382 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:42.383 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:42.400 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:42.400 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:42.412 INFO:tasks.workunit.client.0.smithi031.stdout:rbd journal '1286a02efebc': 2024-09-21T23:26:42.412 INFO:tasks.workunit.client.0.smithi031.stdout: header_oid: journal.1286a02efebc 2024-09-21T23:26:42.412 INFO:tasks.workunit.client.0.smithi031.stdout: object_oid_prefix: journal_data.2.1286a02efebc. 2024-09-21T23:26:42.412 INFO:tasks.workunit.client.0.smithi031.stdout: order: 24 (16 MiB objects) 2024-09-21T23:26:42.413 INFO:tasks.workunit.client.0.smithi031.stdout: splay_width: 4 2024-09-21T23:26:42.416 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal status 1286a02efebc 2024-09-21T23:26:42.419 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:42.419 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:42.436 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:42.436 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:42.441 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:42.440+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:42.451 INFO:tasks.workunit.client.0.smithi031.stdout:minimum_set: 0 2024-09-21T23:26:42.452 INFO:tasks.workunit.client.0.smithi031.stdout:active_set: 0 2024-09-21T23:26:42.452 INFO:tasks.workunit.client.0.smithi031.stdout:registered clients: 2024-09-21T23:26:42.452 INFO:tasks.workunit.client.0.smithi031.stdout: [id=, commit_position=[positions=[]], state=connected] 2024-09-21T23:26:42.455 INFO:tasks.workunit.client.0.smithi031.stderr:+ local count=10 2024-09-21T23:26:42.455 INFO:tasks.workunit.client.0.smithi031.stderr:+ save_commit_position 1286a02efebc 2024-09-21T23:26:42.455 INFO:tasks.workunit.client.0.smithi031.stderr:+ local journal=1286a02efebc 2024-09-21T23:26:42.455 INFO:tasks.workunit.client.0.smithi031.stderr:+ rados -p rbd getomapval journal.1286a02efebc client_ /tmp/rbd_journal46970/1286a02efebc.client_.omap 2024-09-21T23:26:42.455 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:42.456 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:42.490 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:42.490 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:42.496 INFO:tasks.workunit.client.0.smithi031.stderr:Writing to /tmp/rbd_journal46970/1286a02efebc.client_.omap 2024-09-21T23:26:42.499 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd bench --io-type write testrbdjournal46970 --io-size 4096 --io-threads 1 --io-total 40960 --io-pattern seq 2024-09-21T23:26:42.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:42.541 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:42.540+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:42.643 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:42.642+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:42.648 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:42.648 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:42.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:42.744 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:42.743+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:42.763 INFO:tasks.workunit.client.0.smithi031.stdout:bench type write io_size 4096 io_threads 1 bytes 40960 pattern sequential 2024-09-21T23:26:42.776 INFO:tasks.workunit.client.0.smithi031.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-21T23:26:42.777 INFO:tasks.workunit.client.0.smithi031.stdout:elapsed: 0 ops: 10 ops/sec: 769.243 bytes/sec: 3.0 MiB/s 2024-09-21T23:26:42.787 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal status --image testrbdjournal46970 2024-09-21T23:26:42.787 INFO:tasks.workunit.client.0.smithi031.stderr:+ fgrep tid=9 2024-09-21T23:26:42.809 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:42.809 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:42.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:42.840 INFO:tasks.workunit.client.0.smithi031.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=11], [object_number=2, tag_tid=1, entry_tid=10], [object_number=1, tag_tid=1, entry_tid=9], [object_number=0, tag_tid=1, entry_tid=8]]], state=connected] 2024-09-21T23:26:42.841 INFO:tasks.workunit.client.0.smithi031.stderr:+ restore_commit_position 1286a02efebc 2024-09-21T23:26:42.841 INFO:tasks.workunit.client.0.smithi031.stderr:+ local journal=1286a02efebc 2024-09-21T23:26:42.841 INFO:tasks.workunit.client.0.smithi031.stderr:+ rados -p rbd setomapval journal.1286a02efebc client_ 2024-09-21T23:26:42.845 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:42.844+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:42.880 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal status --image testrbdjournal46970 2024-09-21T23:26:42.880 INFO:tasks.workunit.client.0.smithi031.stderr:+ fgrep 'positions=[]' 2024-09-21T23:26:42.933 INFO:tasks.workunit.client.0.smithi031.stdout: [id=, commit_position=[positions=[]], state=connected] 2024-09-21T23:26:42.934 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd journal inspect --verbose 1286a02efebc 2024-09-21T23:26:42.934 INFO:tasks.workunit.client.0.smithi031.stderr:++ grep -c 'event_type.*AioWrite' 2024-09-21T23:26:42.946 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:42.946+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:42.952 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:42.952 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:42.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:42.980 INFO:tasks.workunit.client.0.smithi031.stderr:2024-09-21T23:26:42.979+0000 7fbe1d776640 -1 JournalMetadata: 0x55d85015ec00 failed to locate client: INSPECT 2024-09-21T23:26:42.983 INFO:tasks.workunit.client.0.smithi031.stderr:+ local count1=10 2024-09-21T23:26:42.983 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 10 -eq 10 2024-09-21T23:26:42.983 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal export 1286a02efebc /tmp/rbd_journal46970/journal.export 2024-09-21T23:26:43.025 INFO:tasks.workunit.client.0.smithi031.stdout:12 entries processed, 0 errors 2024-09-21T23:26:43.031 INFO:tasks.workunit.client.0.smithi031.stderr:2024-09-21T23:26:43.031+0000 7f658e55e640 -1 JournalMetadata: 0x56229895ede0 failed to locate client: EXPORT 2024-09-21T23:26:43.034 INFO:tasks.workunit.client.0.smithi031.stderr:++ stat -c %s /tmp/rbd_journal46970/journal.export 2024-09-21T23:26:43.035 INFO:tasks.workunit.client.0.smithi031.stderr:+ local size=55926 2024-09-21T23:26:43.035 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 55926 -gt 0 2024-09-21T23:26:43.035 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd export testrbdjournal46970 /tmp/rbd_journal46970/testrbdjournal46970.export 2024-09-21T23:26:43.046 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:43.046+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:43.085 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:43.085 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:43.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:43.103 INFO:tasks.workunit.client.0.smithi031.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-21T23:26:43.108 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image1=testrbdjournal469701 2024-09-21T23:26:43.108 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --image-feature exclusive-lock --image-feature journaling --size 128 testrbdjournal469701 2024-09-21T23:26:43.147 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:43.146+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:43.166 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd info testrbdjournal469701 --format=xml 2024-09-21T23:26:43.166 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //image/journal 2024-09-21T23:26:43.203 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:43.204 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:43.210 INFO:tasks.workunit.client.0.smithi031.stderr:+ journal1=13013c9ae692 2024-09-21T23:26:43.210 INFO:tasks.workunit.client.0.smithi031.stderr:+ save_commit_position 13013c9ae692 2024-09-21T23:26:43.211 INFO:tasks.workunit.client.0.smithi031.stderr:+ local journal=13013c9ae692 2024-09-21T23:26:43.211 INFO:tasks.workunit.client.0.smithi031.stderr:+ rados -p rbd getomapval journal.13013c9ae692 client_ /tmp/rbd_journal46970/13013c9ae692.client_.omap 2024-09-21T23:26:43.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:43.246 INFO:tasks.workunit.client.0.smithi031.stderr:Writing to /tmp/rbd_journal46970/13013c9ae692.client_.omap 2024-09-21T23:26:43.249 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal import --dest testrbdjournal469701 /tmp/rbd_journal46970/journal.export 2024-09-21T23:26:43.255 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:43.254+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:43.306 INFO:tasks.workunit.client.0.smithi031.stdout:12 entries processed, 0 errors 2024-09-21T23:26:43.306 INFO:tasks.workunit.client.0.smithi031.stdout:Waiting for journal append to complete... 2024-09-21T23:26:43.318 INFO:tasks.workunit.client.0.smithi031.stderr:2024-09-21T23:26:43.317+0000 7fb1127fc640 -1 JournalMetadata: 0x56511c9fccc0 failed to locate client: IMPORT 2024-09-21T23:26:43.321 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd snap create testrbdjournal469701@test 2024-09-21T23:26:43.349 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:43.348+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:43.363 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:43.363 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:43.373 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:43.450 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:43.450+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:43.520 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:43.521 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:43.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:43.573 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:43.572+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:43.637 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:43.637 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:43.646 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:43.651 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:43.651+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:43.752 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:43.751+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:43.794 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:43.795 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:43.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:43.887 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:43.886+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:43.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T23:26:43.899 INFO:tasks.workunit.client.0.smithi031.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T23:26:43.909 INFO:tasks.workunit.client.0.smithi031.stderr:+ restore_commit_position 13013c9ae692 2024-09-21T23:26:43.909 INFO:tasks.workunit.client.0.smithi031.stderr:+ local journal=13013c9ae692 2024-09-21T23:26:43.909 INFO:tasks.workunit.client.0.smithi031.stderr:+ rados -p rbd setomapval journal.13013c9ae692 client_ 2024-09-21T23:26:43.953 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:43.953 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:43.953 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:43.952+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:43.954 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal inspect --image testrbdjournal469701 --verbose 2024-09-21T23:26:43.954 INFO:tasks.workunit.client.0.smithi031.stderr:+ awk ' 2024-09-21T23:26:43.954 INFO:tasks.workunit.client.0.smithi031.stderr: /AioFlush/ {a++} # match: "event_type": "AioFlush", 2024-09-21T23:26:43.954 INFO:tasks.workunit.client.0.smithi031.stderr: /AioWrite/ {w++} # match: "event_type": "AioWrite", 2024-09-21T23:26:43.954 INFO:tasks.workunit.client.0.smithi031.stderr: /SnapCreate/ {s++} # match: "event_type": "SnapCreate", 2024-09-21T23:26:43.954 INFO:tasks.workunit.client.0.smithi031.stderr: /OpFinish/ {f++} # match: "event_type": "OpFinish", 2024-09-21T23:26:43.954 INFO:tasks.workunit.client.0.smithi031.stderr: /entries inspected/ {t=$1; e=$4} # match: 14 entries inspected, 0 errors 2024-09-21T23:26:43.954 INFO:tasks.workunit.client.0.smithi031.stderr: {print} # for diagnostic 2024-09-21T23:26:43.954 INFO:tasks.workunit.client.0.smithi031.stderr: END { 2024-09-21T23:26:43.954 INFO:tasks.workunit.client.0.smithi031.stderr: if (a != 2 || w != 10 || s != 1 || f != 1 || t != 14 || e != 0) exit(1) 2024-09-21T23:26:43.954 INFO:tasks.workunit.client.0.smithi031.stderr: } 2024-09-21T23:26:43.955 INFO:tasks.workunit.client.0.smithi031.stderr: ' 2024-09-21T23:26:43.962 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:44.019 INFO:tasks.workunit.client.0.smithi031.stderr:2024-09-21T23:26:44.019+0000 7f4e937fe640 -1 JournalMetadata: 0x55c76b5e8940 failed to locate client: INSPECT 2024-09-21T23:26:44.022 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=1 2024-09-21T23:26:44.022 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioFlush", 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.762217+0000" 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=2 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioWrite", 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout: "offset": 0, 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout: "length": 4096, 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.764272+0000" 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=3 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioWrite", 2024-09-21T23:26:44.023 INFO:tasks.workunit.client.0.smithi031.stdout: "offset": 4096, 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout: "length": 4096, 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.764337+0000" 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=4 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioWrite", 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout: "offset": 8192, 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout: "length": 4096, 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.764381+0000" 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=5 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioWrite", 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout: "offset": 12288, 2024-09-21T23:26:44.024 INFO:tasks.workunit.client.0.smithi031.stdout: "length": 4096, 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.764434+0000" 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=6 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioWrite", 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout: "offset": 16384, 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout: "length": 4096, 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.764481+0000" 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=7 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioWrite", 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout: "offset": 20480, 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout: "length": 4096, 2024-09-21T23:26:44.025 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.764518+0000" 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=8 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioWrite", 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout: "offset": 24576, 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout: "length": 4096, 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.764576+0000" 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=9 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioWrite", 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout: "offset": 28672, 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout: "length": 4096, 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.764610+0000" 2024-09-21T23:26:44.026 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=10 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioWrite", 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout: "offset": 32768, 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout: "length": 4096, 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.764647+0000" 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=11 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioWrite", 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout: "offset": 36864, 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout: "length": 4096, 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.764681+0000" 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.027 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=12 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "AioFlush", 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:42.764715+0000" 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=13 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "SnapCreate", 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout: "op_tid": 1, 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout: "snap_name": "test", 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout: "snapshot_namespace_type": "user", 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:43.382634+0000" 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout:Entry: tag_id=1, commit_tid=14 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout:{ 2024-09-21T23:26:44.028 INFO:tasks.workunit.client.0.smithi031.stdout: "event_type": "OpFinish", 2024-09-21T23:26:44.029 INFO:tasks.workunit.client.0.smithi031.stdout: "op_tid": 1, 2024-09-21T23:26:44.029 INFO:tasks.workunit.client.0.smithi031.stdout: "op_tid": 1, 2024-09-21T23:26:44.029 INFO:tasks.workunit.client.0.smithi031.stdout: "result": 0, 2024-09-21T23:26:44.029 INFO:tasks.workunit.client.0.smithi031.stdout: "timestamp": "2024-09-21T23:26:43.895011+0000" 2024-09-21T23:26:44.029 INFO:tasks.workunit.client.0.smithi031.stdout:} 2024-09-21T23:26:44.029 INFO:tasks.workunit.client.0.smithi031.stdout:Summary: 2024-09-21T23:26:44.029 INFO:tasks.workunit.client.0.smithi031.stdout: 14 entries inspected, 0 errors 2024-09-21T23:26:44.029 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd export testrbdjournal469701@test /tmp/rbd_journal46970/testrbdjournal469701.export 2024-09-21T23:26:44.054 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:44.054+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:44.091 INFO:tasks.workunit.client.0.smithi031.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-21T23:26:44.097 INFO:tasks.workunit.client.0.smithi031.stderr:+ cmp /tmp/rbd_journal46970/testrbdjournal46970.export /tmp/rbd_journal46970/testrbdjournal469701.export 2024-09-21T23:26:44.111 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:44.111 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:44.120 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:44.164 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:44.163+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:44.194 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal reset 1286a02efebc 2024-09-21T23:26:44.227 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:44.227 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:44.237 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:44.256 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:44.256+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:44.266 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd journal inspect --verbose 1286a02efebc 2024-09-21T23:26:44.266 INFO:tasks.workunit.client.0.smithi031.stderr:+ expect_false grep event_type 2024-09-21T23:26:44.266 INFO:tasks.workunit.client.0.smithi031.stderr:+ set -x 2024-09-21T23:26:44.266 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep event_type 2024-09-21T23:26:44.308 INFO:tasks.workunit.client.0.smithi031.stderr:2024-09-21T23:26:44.308+0000 7efccb7fe640 -1 JournalMetadata: 0x56529d4d1450 failed to locate client: INSPECT 2024-09-21T23:26:44.311 INFO:tasks.workunit.client.0.smithi031.stderr:+ return 0 2024-09-21T23:26:44.311 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd snap purge testrbdjournal469701 2024-09-21T23:26:44.357 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:44.356+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:44.389 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:44.389 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:44.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:44.458 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:44.458+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:44.471 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:26:44.505 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:44.505 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:44.513 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:44.558 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:44.558+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:44.595 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:44.595 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:44.612 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:44.612 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:44.620 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:44.620 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:44.628 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:44.628 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:44.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:44.645 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:44.645 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:44.659 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:44.658+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:44.662 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:44.663 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:44.696 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:44.696 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:44.713 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:44.713 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:44.730 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:44.731 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:44.749 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:44.749 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:44.749 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:44.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:44.799 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:44.799+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:44.862 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:44.862+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:44.888 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-09-21T23:26:44.902 INFO:tasks.workunit.client.0.smithi031.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-21T23:26:44.904 INFO:tasks.workunit.client.0.smithi031.stderr:2024-09-21T23:26:44.904+0000 7f35d265a640 0 -- 172.21.15.31:0/3352402684 >> [v2:172.21.15.31:6808/864917651,v1:172.21.15.31:6809/864917651] conn(0x7f35b4071b20 msgr2=0x7f35b4073f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:44.910 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd remove testrbdjournal469701 2024-09-21T23:26:44.912 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:44.913 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:44.913 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:44.922 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:44.963 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:44.962+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:44.993 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete...2024-09-21T23:26:44.993+0000 7f6b0c7b3640 0 -- 172.21.15.31:0/3853830039 >> [v2:172.21.15.60:6800/3490482162,v1:172.21.15.60:6801/3490482162] conn(0x7f6ae806f840 msgr2=0x7f6ae806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:45.011 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-09-21T23:26:45.011+0000 7f6b0da15640 0 -- 172.21.15.31:0/3853830039 >> [v2:172.21.15.31:6824/2889649822,v1:172.21.15.31:6825/2889649822] conn(0x55ae28e45ae0 msgr2=0x55ae28e47ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:45.063 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:45.063+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:45.078 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:45.078 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.940730349999995, 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036127939999999999, 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:45.079 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136601+0000", 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3500000000000002e-07 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.080 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136608+0000", 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.182e-06 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136618+0000", 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9199999999999999e-06 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136628+0000", 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4390000000000006e-06 2024-09-21T23:26:45.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136692+0000", 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3979000000000005e-05 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.136716+0000", 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4391000000000001e-05 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.082 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140171+0000", 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034554910000000002 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.140214+0000", 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2057000000000001e-05 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.083 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'18)", 2024-09-21T23:26:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.906212186999994, 2024-09-21T23:26:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043957730000000004, 2024-09-21T23:26:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:45.084 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171119+0000", 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171120+0000", 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6700000000000003e-07 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171125+0000", 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.451e-06 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171138+0000", 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2541000000000001e-05 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171143+0000", 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4659999999999998e-06 2024-09-21T23:26:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171187+0000", 2024-09-21T23:26:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4014000000000002e-05 2024-09-21T23:26:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.171200+0000", 2024-09-21T23:26:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3108999999999999e-05 2024-09-21T23:26:45.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175479+0000", 2024-09-21T23:26:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042784570000000003 2024-09-21T23:26:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.090 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.175515+0000", 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.5967999999999999e-05 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:58 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.883459891000001, 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003032851, 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193871+0000", 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193872+0000", 2024-09-21T23:26:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9400000000000004e-07 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193952+0000", 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9791999999999998e-05 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193962+0000", 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0034000000000001e-05 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.093 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.193973+0000", 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0733e-05 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194015+0000", 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2911999999999998e-05 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194031+0000", 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5082999999999999e-05 2024-09-21T23:26:45.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196869+0000", 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002838808 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.196904+0000", 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4694999999999999e-05 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.095 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:59 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-21T23:26:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.882874224999995, 2024-09-21T23:26:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053675709999999998, 2024-09-21T23:26:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.096 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194457+0000", 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7100000000000002e-07 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.097 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194462+0000", 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.668e-06 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194469+0000", 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1670000000000002e-06 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194475+0000", 2024-09-21T23:26:45.098 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-06 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194513+0000", 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7391999999999997e-05 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.194531+0000", 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7870999999999999e-05 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.099 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199791+0000", 2024-09-21T23:26:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052604119999999999 2024-09-21T23:26:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.199824+0000", 2024-09-21T23:26:45.100 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3340000000000003e-05 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:60 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'38, mlcod=16'36)", 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.87679353, 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003614562, 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200538+0000", 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200539+0000", 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5799999999999999e-06 2024-09-21T23:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200551+0000", 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1816e-05 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200578+0000", 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6772999999999999e-05 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200590+0000", 2024-09-21T23:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2303000000000001e-05 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200715+0000", 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012523499999999999 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.200731+0000", 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5690000000000001e-05 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.104 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204121+0000", 2024-09-21T23:26:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003390363 2024-09-21T23:26:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.204152+0000", 2024-09-21T23:26:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0802000000000001e-05 2024-09-21T23:26:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:70 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'39)", 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.866737874999998, 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033896579999999998, 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210593+0000", 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210594+0000", 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1099999999999995e-07 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210600+0000", 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4389999999999997e-06 2024-09-21T23:26:45.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210613+0000", 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.243e-05 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210618+0000", 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0939999999999998e-06 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210843+0000", 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00022493200000000001 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.210858+0000", 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5109000000000001e-05 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213938+0000", 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030796500000000002 2024-09-21T23:26:45.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.213983+0000", 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5293000000000002e-05 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:72 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'46, mlcod=16'43)", 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:45.110 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.861793098999996, 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029102279999999999, 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.111 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215538+0000", 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215539+0000", 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4199999999999995e-07 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215589+0000", 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0488999999999997e-05 2024-09-21T23:26:45.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215598+0000", 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2830000000000007e-06 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215610+0000", 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1134999999999999e-05 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215657+0000", 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7658999999999999e-05 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.215679+0000", 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1526999999999999e-05 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218410+0000", 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027316939999999998 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.218448+0000", 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7799000000000002e-05 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:73 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'47, mlcod=16'43)", 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.861183752000002, 2024-09-21T23:26:45.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051882020000000003, 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216147+0000", 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216148+0000", 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7100000000000001e-07 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216218+0000", 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0313000000000003e-05 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216227+0000", 2024-09-21T23:26:45.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9959999999999999e-06 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216232+0000", 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5340000000000001e-06 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216267+0000", 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4739000000000001e-05 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216281+0000", 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4345000000000001e-05 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221290+0000", 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050088989999999998 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221336+0000", 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5704999999999997e-05 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:74 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'48, mlcod=16'43)", 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.860657809000003, 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046922939999999996, 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216673+0000", 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216674+0000", 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9900000000000001e-07 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216679+0000", 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.502e-06 2024-09-21T23:26:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216687+0000", 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1790000000000008e-06 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216691+0000", 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7230000000000002e-06 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216737+0000", 2024-09-21T23:26:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6087000000000001e-05 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.216758+0000", 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0434000000000001e-05 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221338+0000", 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045804569999999996 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.221366+0000", 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7413e-05 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:86 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'56, mlcod=16'50)", 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.843235875000005, 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049055269999999998, 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234095+0000", 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234096+0000", 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2699999999999999e-07 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234099+0000", 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3890000000000001e-06 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234107+0000", 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2349999999999999e-06 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234112+0000", 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0629999999999999e-06 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234143+0000", 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.1131999999999999e-05 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234155+0000", 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2140000000000001e-05 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.238972+0000", 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048167770000000004 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.239001+0000", 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9164000000000001e-05 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:87 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'57, mlcod=16'52)", 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.842556500000001, 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0067179910000000004, 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234775+0000", 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0299999999999998e-07 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234813+0000", 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7531999999999999e-05 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234826+0000", 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2617e-05 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234831+0000", 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1229999999999999e-06 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234869+0000", 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.8513999999999998e-05 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.234881+0000", 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2148e-05 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241465+0000", 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065837329999999996 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241493+0000", 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7521e-05 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.132 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:88 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.841758651999996, 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059344530000000001, 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235572+0000", 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235573+0000", 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.0100000000000004e-07 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235578+0000", 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1590000000000001e-06 2024-09-21T23:26:45.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235590+0000", 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1229e-05 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235595+0000", 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0889999999999999e-06 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235943+0000", 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000348227 2024-09-21T23:26:45.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.235957+0000", 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4419999999999999e-05 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241494+0000", 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055371070000000003 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.241507+0000", 2024-09-21T23:26:45.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2520999999999999e-05 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:113 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'75, mlcod=16'71)", 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.682527374000003, 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028543489999999999, 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394804+0000", 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8199999999999999e-07 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394809+0000", 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0690000000000001e-06 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394821+0000", 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2024999999999999e-05 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394827+0000", 2024-09-21T23:26:45.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0020000000000003e-06 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394862+0000", 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4811000000000003e-05 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.394873+0000", 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.058e-05 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397631+0000", 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027581060000000002 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397658+0000", 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7073999999999999e-05 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'71)", 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:45.141 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.682115960999994, 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005160085, 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395215+0000", 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395216+0000", 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4999999999999998e-07 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395221+0000", 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2290000000000002e-06 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395229+0000", 2024-09-21T23:26:45.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2369999999999992e-06 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395233+0000", 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1840000000000001e-06 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395260+0000", 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.728e-05 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395272+0000", 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1270000000000001e-05 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400341+0000", 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050691809999999999 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400375+0000", 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4254000000000001e-05 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'72)", 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.681567439999995, 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046274580000000001, 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395764+0000", 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2e-07 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395829+0000", 2024-09-21T23:26:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4618999999999999e-05 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395836+0000", 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9090000000000003e-06 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395839+0000", 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.371e-06 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395868+0000", 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9328000000000001e-05 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.395880+0000", 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1501000000000001e-05 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400376+0000", 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044966340000000002 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400391+0000", 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4676e-05 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:116 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'78, mlcod=16'73)", 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.680762892999994, 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038400970000000002, 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396568+0000", 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396569+0000", 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1100000000000001e-07 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396573+0000", 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5569999999999997e-06 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396579+0000", 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1120000000000001e-06 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396587+0000", 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396630+0000", 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2991000000000002e-05 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.396642+0000", 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1481000000000001e-05 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400392+0000", 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037507830000000002 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400408+0000", 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5863e-05 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:117 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'79, mlcod=16'73)", 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.680260551000003, 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033512120000000001, 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397071+0000", 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7e-07 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397075+0000", 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7809999999999999e-06 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397082+0000", 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9280000000000004e-06 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397085+0000", 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7139999999999999e-06 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397130+0000", 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.498e-05 2024-09-21T23:26:45.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.397142+0000", 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1838000000000001e-05 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400409+0000", 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032666349999999999 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.400422+0000", 2024-09-21T23:26:45.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2866e-05 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.675436877999999, 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034761240000000001, 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401894+0000", 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401895+0000", 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7400000000000003e-07 2024-09-21T23:26:45.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401963+0000", 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8701000000000005e-05 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401972+0000", 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9530000000000005e-06 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.401977+0000", 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.1579999999999998e-06 2024-09-21T23:26:45.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402009+0000", 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2512000000000003e-05 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402022+0000", 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3220999999999999e-05 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405337+0000", 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033150300000000001 2024-09-21T23:26:45.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.405370+0000", 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2975000000000002e-05 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.674830807999996, 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054168369999999999, 2024-09-21T23:26:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402500+0000", 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402501+0000", 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2699999999999999e-07 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402589+0000", 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7899000000000007e-05 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402599+0000", 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0312000000000001e-05 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402605+0000", 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9689999999999999e-06 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402648+0000", 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3210000000000001e-05 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.402664+0000", 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5347999999999999e-05 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407887+0000", 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052237919999999997 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407917+0000", 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9779999999999999e-05 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-09-21T23:26:45.167 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 83.674301427000003, 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049049180000000003, 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403030+0000", 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4099999999999999e-07 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403098+0000", 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7464000000000003e-05 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403106+0000", 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6629999999999999e-06 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403111+0000", 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403137+0000", 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.6349000000000001e-05 2024-09-21T23:26:45.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.403148+0000", 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.077e-05 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407920+0000", 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047716590000000001 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:21.407935+0000", 2024-09-21T23:26:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4783999999999999e-05 2024-09-21T23:26:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.172 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.172 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:45.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:45.174 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:45.172+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:45.222 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 100% complete...done. 2024-09-21T23:26:45.225 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd remove testrbdjournal46970 2024-09-21T23:26:45.274 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:45.273+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:45.309 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete...2024-09-21T23:26:45.308+0000 7fa32b70a640 0 -- 172.21.15.31:0/3823599977 >> [v2:172.21.15.31:6808/864917651,v1:172.21.15.31:6809/864917651] conn(0x7fa308016840 msgr2=0x7fa3080152d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:45.313 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 962024-09-21T23:26:45.312+0000 7fa32bf0b640 0 -- 172.21.15.31:0/3823599977 >> [v2:172.21.15.60:6824/2320687649,v1:172.21.15.60:6825/2320687649] conn(0x55890ea6f430 msgr2=0x55890ea718c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:45.333 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:45.333 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:45.333 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:45.340 INFO:tasks.workunit.client.0.smithi031.stderr:% complete... Removing image: 100% complete...done. 2024-09-21T23:26:45.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:45.344 INFO:tasks.workunit.client.0.smithi031.stderr:+ set +x 2024-09-21T23:26:45.344 INFO:tasks.workunit.client.0.smithi031.stderr:+ test_rbd_create 2024-09-21T23:26:45.344 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdcreate46970 2024-09-21T23:26:45.344 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --image-feature exclusive-lock --image-feature journaling --journal-pool rbd --journal-object-size 20M --journal-splay-width 6 --size 256 testrbdcreate46970 2024-09-21T23:26:45.375 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:45.374+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:45.407 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdcreate46970 'journal info' //journal/order 25 2024-09-21T23:26:45.407 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdcreate46970 2024-09-21T23:26:45.407 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:45.408 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/order 2024-09-21T23:26:45.408 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=25 2024-09-21T23:26:45.408 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdcreate46970 2024-09-21T23:26:45.408 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/order 2024-09-21T23:26:45.455 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:45.456 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:45.456 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:45.456 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:45.456 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:45.460 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=25 2024-09-21T23:26:45.460 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 25 = 25 2024-09-21T23:26:45.460 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdcreate46970 'journal info' //journal/splay_width 6 2024-09-21T23:26:45.460 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdcreate46970 2024-09-21T23:26:45.460 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:45.461 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/splay_width 2024-09-21T23:26:45.461 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=6 2024-09-21T23:26:45.461 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdcreate46970 2024-09-21T23:26:45.461 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/splay_width 2024-09-21T23:26:45.465 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:45.475 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:45.474+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:45.515 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=6 2024-09-21T23:26:45.515 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 6 = 6 2024-09-21T23:26:45.515 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdcreate46970 'journal info' //journal/object_pool rbd 2024-09-21T23:26:45.515 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdcreate46970 2024-09-21T23:26:45.515 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:45.515 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/object_pool 2024-09-21T23:26:45.515 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=rbd 2024-09-21T23:26:45.515 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdcreate46970 2024-09-21T23:26:45.516 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/object_pool 2024-09-21T23:26:45.570 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=rbd 2024-09-21T23:26:45.570 INFO:tasks.workunit.client.0.smithi031.stderr:+ test rbd = rbd 2024-09-21T23:26:45.571 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd remove testrbdcreate46970 2024-09-21T23:26:45.576 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:45.575+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:45.580 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:45.580 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 76.408490595000004, 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003527526, 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.581 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170742+0000", 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170743+0000", 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.08e-07 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170750+0000", 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7800000000000001e-06 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170767+0000", 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6923999999999999e-05 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170788+0000", 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0446e-05 2024-09-21T23:26:45.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170849+0000", 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1409999999999996e-05 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.170881+0000", 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2075e-05 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174218+0000", 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033371070000000002 2024-09-21T23:26:45.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:29.174270+0000", 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1375999999999997e-05 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:73 2.4 2:28a01f35:::rbd_data.13013c9ae692.0000000000000002:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.993648+0000", 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.58558529199999998, 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048601310000000002, 2024-09-21T23:26:45.585 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 73 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993648+0000", 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.586 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993648+0000", 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993649+0000", 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.728e-06 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993658+0000", 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5010000000000004e-06 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993677+0000", 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9122000000000001e-05 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993689+0000", 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1752e-05 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993770+0000", 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1304999999999995e-05 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993900+0000", 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013050099999999999 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998508+0000", 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046072220000000002 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:78 2.4 2:2f3e3e19:::rbd_data.13013c9ae692.0000000000000007:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:45.589 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.993772+0000", 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.58546057299999998, 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0071295330000000004, 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 78 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.590 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993772+0000", 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993772+0000", 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993773+0000", 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1099999999999995e-07 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.591 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993797+0000", 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4490000000000001e-05 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993807+0000", 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3749999999999992e-06 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993813+0000", 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4350000000000002e-06 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994193+0000", 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00038006500000000001 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994310+0000", 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116495 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.000902+0000", 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065920620000000001 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.593 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:80 2.4 2:28f58da5:::rbd_data.13013c9ae692.0000000000000009:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.993829+0000", 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.58540416699999998, 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0071714470000000001, 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:45.594 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 80 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993829+0000", 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993829+0000", 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993829+0000", 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6400000000000002e-07 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993834+0000", 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2570000000000001e-06 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993841+0000", 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2930000000000003e-06 2024-09-21T23:26:45.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993846+0000", 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0860000000000001e-06 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994567+0000", 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00072146099999999996 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994673+0000", 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106118 2024-09-21T23:26:45.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001000+0000", 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0063266679999999997 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.5259.0:71 2.2 e57/18 2:52b8847e:::rbd_data.13013c9ae692.0000000000000000:head v 57'52, mlcod=57'49)", 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.993960+0000", 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.58527323499999995, 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003614824, 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.598 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993960+0000", 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993960+0000", 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.599 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993961+0000", 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.099e-06 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993969+0000", 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5320000000000006e-06 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993991+0000", 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1480000000000001e-05 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.600 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993998+0000", 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7200000000000006e-06 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994052+0000", 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3316000000000002e-05 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994071+0000", 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8978999999999999e-05 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.601 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997531+0000", 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034600569999999999 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997574+0000", 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3640999999999998e-05 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.0 epoch 57/18 rep_tid 164 entries 57'1 (0'0) error 2:15bd28b2:::rbd_data.13013c9ae692.0000000000000004:head by client.5259.0:75 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-21T23:26:45.602 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.994163+0000", 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.58506938900000005, 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003373666, 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994163+0000", 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994163+0000", 2024-09-21T23:26:45.603 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994164+0000", 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.03e-07 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994171+0000", 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9569999999999996e-06 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994186+0000", 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5608e-05 2024-09-21T23:26:45.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994200+0000", 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3691e-05 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994288+0000", 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7825999999999996e-05 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997537+0000", 2024-09-21T23:26:45.605 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032491809999999999 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:83 2.4 2:2574ce61:::rbd_data.13013c9ae692.000000000000000c:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.997889+0000", 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.58134378399999997, 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057942690000000003, 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.606 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 83 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997889+0000", 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997889+0000", 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997890+0000", 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.246e-06 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997899+0000", 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1960000000000008e-06 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997909+0000", 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0869e-05 2024-09-21T23:26:45.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997929+0000", 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9565999999999999e-05 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997982+0000", 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2680999999999999e-05 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998118+0000", 2024-09-21T23:26:45.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013619399999999999 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003683+0000", 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055655169999999999 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:86 2.4 2:3fcce9c5:::rbd_data.13013c9ae692.000000000000000f:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.998646+0000", 2024-09-21T23:26:45.610 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.58058677000000003, 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051920330000000004, 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 86 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998646+0000", 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998646+0000", 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998647+0000", 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7400000000000002e-07 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998652+0000", 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998663+0000", 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1438e-05 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998671+0000", 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9950000000000005e-06 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998727+0000", 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5856000000000002e-05 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998858+0000", 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013130699999999999 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003838+0000", 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049797749999999997 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.614 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.2 epoch 57/18 rep_tid 65 entries 57'53 (0'0) error 2:4dd0ab61:::rbd_data.13013c9ae692.0000000000000013:head by client.5259.0:90 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'51)", 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.001575+0000", 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.57765836199999998, 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046952690000000002, 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001575+0000", 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001575+0000", 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001575+0000", 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-07 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001583+0000", 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5739999999999998e-06 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001598+0000", 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5123000000000001e-05 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001607+0000", 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3589999999999997e-06 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001685+0000", 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8176000000000005e-05 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006270+0000", 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045844220000000003 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:92 2.4 2:295f8ccd:::rbd_data.13013c9ae692.0000000000000015:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.003886+0000", 2024-09-21T23:26:45.618 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.57534638699999996, 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051605130000000003, 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 92 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003886+0000", 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003886+0000", 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003887+0000", 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.018e-06 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003892+0000", 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.831e-06 2024-09-21T23:26:45.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003911+0000", 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8564e-05 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003919+0000", 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6000000000000007e-06 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003982+0000", 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2848999999999994e-05 2024-09-21T23:26:45.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004097+0000", 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000114855 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009047+0000", 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049497960000000002 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:95 2.4 2:245a1e52:::rbd_data.13013c9ae692.0000000000000018:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.004138+0000", 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.57509513800000001, 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049644850000000003, 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 95 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004138+0000", 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004138+0000", 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004139+0000", 2024-09-21T23:26:45.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2600000000000001e-07 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004143+0000", 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4290000000000003e-06 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004153+0000", 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0261000000000001e-05 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004161+0000", 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2830000000000004e-06 2024-09-21T23:26:45.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004346+0000", 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018584299999999999 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004470+0000", 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012406900000000001 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009102+0000", 2024-09-21T23:26:45.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004631774 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.0 epoch 57/18 rep_tid 168 entries 57'2 (0'0) error 2:197c199e:::rbd_data.13013c9ae692.0000000000000017:head by client.5259.0:94 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.004388+0000", 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.57484475000000002, 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046250680000000004, 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004388+0000", 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004388+0000", 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004389+0000", 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4000000000000001e-07 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004397+0000", 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1149999999999994e-06 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004414+0000", 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7575e-05 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004432+0000", 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7655999999999998e-05 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004513+0000", 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0482999999999995e-05 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009013+0000", 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045005990000000001 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.630 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.2 epoch 57/18 rep_tid 67 entries 57'54 (0'0) error 2:4a2d9a29:::rbd_data.13013c9ae692.0000000000000019:head by client.5259.0:96 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'51)", 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.004588+0000", 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.57464513900000003, 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044264439999999999, 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004588+0000", 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004588+0000", 2024-09-21T23:26:45.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004588+0000", 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7400000000000003e-07 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004595+0000", 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7560000000000002e-06 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004610+0000", 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4693e-05 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004631+0000", 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.084e-05 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004702+0000", 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1222000000000004e-05 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009014+0000", 2024-09-21T23:26:45.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004312359 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:100 2.4 2:3c2bac98:::rbd_data.13013c9ae692.000000000000001d:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.006738+0000", 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.57249482200000001, 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053749649999999998, 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.634 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 100 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006738+0000", 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006738+0000", 2024-09-21T23:26:45.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006739+0000", 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6300000000000004e-07 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006742+0000", 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3230000000000002e-06 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006752+0000", 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.329e-06 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006756+0000", 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6609999999999999e-06 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006815+0000", 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8683e-05 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.637 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006937+0000", 2024-09-21T23:26:45.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000122309 2024-09-21T23:26:45.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.012113+0000", 2024-09-21T23:26:45.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051757970000000002 2024-09-21T23:26:45.638 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.638 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.638 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.639 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.0 epoch 57/18 rep_tid 171 entries 57'3 (0'0) error 2:11991a95:::rbd_data.13013c9ae692.000000000000001f:head by client.5259.0:102 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'1)", 2024-09-21T23:26:45.639 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.009477+0000", 2024-09-21T23:26:45.639 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.56975559899999995, 2024-09-21T23:26:45.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052640889999999996, 2024-09-21T23:26:45.640 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.640 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:45.640 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.641 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009477+0000", 2024-09-21T23:26:45.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009477+0000", 2024-09-21T23:26:45.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009478+0000", 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.23e-07 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009485+0000", 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1400000000000002e-06 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009501+0000", 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5699999999999999e-05 2024-09-21T23:26:45.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009510+0000", 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0240000000000003e-06 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009568+0000", 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8111e-05 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.014741+0000", 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051733910000000003 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.2 epoch 57/18 rep_tid 68 entries 57'55 (0'0) error 2:58bc8c56:::rbd_data.13013c9ae692.000000000000001e:head by client.5259.0:101 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'53)", 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.009537+0000", 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.56969544400000005, 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052335230000000003, 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:45.645 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009537+0000", 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009537+0000", 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009538+0000", 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.84e-07 2024-09-21T23:26:45.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009541+0000", 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.749e-06 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009555+0000", 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.366e-05 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009563+0000", 2024-09-21T23:26:45.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2230000000000007e-06 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009630+0000", 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6632999999999997e-05 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.014771+0000", 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051409740000000004 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:75 2.4 2:3488d472:::rbd_data.1286a02efebc.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.305141+0000", 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.27409231099999998, 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039025739999999998, 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 75 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.650 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305141+0000", 2024-09-21T23:26:45.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.650 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305141+0000", 2024-09-21T23:26:45.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.651 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305141+0000", 2024-09-21T23:26:45.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4600000000000005e-07 2024-09-21T23:26:45.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305144+0000", 2024-09-21T23:26:45.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3380000000000001e-06 2024-09-21T23:26:45.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305154+0000", 2024-09-21T23:26:45.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0690000000000008e-06 2024-09-21T23:26:45.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305160+0000", 2024-09-21T23:26:45.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3489999999999997e-06 2024-09-21T23:26:45.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305628+0000", 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00046766400000000001 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305742+0000", 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0001145 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.309043+0000", 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033011080000000001 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:97 2.4 2:23779d5c:::rbd_data.1286a02efebc.000000000000001a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.312496+0000", 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.26673734500000001, 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045780710000000004, 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 97 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.656 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312496+0000", 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312496+0000", 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312496+0000", 2024-09-21T23:26:45.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0900000000000001e-07 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312499+0000", 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9560000000000002e-06 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312507+0000", 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.174e-06 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312513+0000", 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7169999999999996e-06 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312992+0000", 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00047860399999999999 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313104+0000", 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000112176 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.317074+0000", 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039698349999999997 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:99 2.4 2:22c0de70:::rbd_data.1286a02efebc.000000000000001c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.312522+0000", 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.26671043100000003, 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046714399999999998, 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.660 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 99 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312522+0000", 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312522+0000", 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312523+0000", 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7100000000000002e-07 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312526+0000", 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9950000000000001e-06 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312533+0000", 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2880000000000004e-06 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312538+0000", 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0479999999999998e-06 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313398+0000", 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00085969099999999999 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313512+0000", 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011357399999999999 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.317194+0000", 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036823730000000001 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:117 2.4 2:3053cb30:::journal_data.2.1286a02efebc.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+full_force+supports_pool_eio e57)", 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.326351+0000", 2024-09-21T23:26:45.664 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.25288206600000002, 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049954220000000002, 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 117 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326351+0000", 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326351+0000", 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326352+0000", 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.009e-06 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:45.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326356+0000", 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3839999999999999e-06 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326367+0000", 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0893999999999999e-05 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326375+0000", 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3189999999999995e-06 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:45.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326428+0000", 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2286e-05 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326475+0000", 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7434e-05 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.331346+0000", 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048710960000000001 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:45.668 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:45.669 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:45.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:45.675 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete...2024-09-21T23:26:45.649+0000 7f6d6696c640 0 -- 172.21.15.31:0/2898988500 >> [v2:172.21.15.31:6824/2889649822,v1:172.21.15.31:6825/2889649822] conn(0x5622eec46e30 msgr2=0x5622eec49220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:45.675 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete...2024-09-21T23:26:45.655+0000 7f6d6f16d640 0 -- 172.21.15.31:0/2898988500 >> [v2:172.21.15.60:6816/1315554966,v1:172.21.15.60:6817/1315554966] conn(0x7f6d48061390 msgr2=0x7f6d48063820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:45.677 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:45.676+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:45.778 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:45.778+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:45.818 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:45.818 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:45.818 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:45.818 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:45.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:45.879 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:45.879+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:45.881 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete... Removing image: 100% complete...done. 2024-09-21T23:26:45.885 INFO:tasks.workunit.client.0.smithi031.stderr:+ set +x 2024-09-21T23:26:45.885 INFO:tasks.workunit.client.0.smithi031.stderr:+ test_rbd_copy 2024-09-21T23:26:45.885 INFO:tasks.workunit.client.0.smithi031.stderr:+ local src=testrbdcopys46970 2024-09-21T23:26:45.885 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 256 testrbdcopys46970 2024-09-21T23:26:45.933 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdcopy46970 2024-09-21T23:26:45.933 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd copy --image-feature exclusive-lock --image-feature journaling --journal-pool rbd --journal-object-size 20M --journal-splay-width 6 testrbdcopys46970 testrbdcopy46970 2024-09-21T23:26:45.941 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:45.941 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:45.941 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:45.941 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:45.942 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:45.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:45.992 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:45.992+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:46.027 INFO:tasks.workunit.client.0.smithi031.stderr: Image copy: 1% complete... Image copy: 3% complete... Image copy: 4% complete... Image copy: 6% complete... Image copy: 7% complete... Image copy: 9% complete... Image copy: 10% complete... Image copy: 12% complete... Image copy: 14% complete... Image copy: 15% complete... Image copy: 17% complete... Image copy: 18% complete... Image copy: 20% complete... Image copy: 21% complete... Image copy: 23% complete... Image copy: 25% complete... Image copy: 26% complete... Image copy: 28% complete... Image copy: 29% complete... Image copy: 31% complete... Image copy: 32% complete... Image copy: 34% complete... Image copy: 35% complete... Image copy: 37% complete... Image copy: 39% complete... Image copy: 40% complete... Image copy: 42% complete... Image copy: 43% complete... Image copy: 45% complete... Image copy: 46% complete... Image copy: 48% complete... Image copy: 50% complete... Image copy: 51% complete... Image copy: 53% complete... Image copy: 54% complete... Image copy: 56% complete... Image copy: 57% complete... Image copy: 59% complete... Image copy: 60% complete... Image copy: 62% complete... Image copy: 64% complete... Image copy: 65% complete... Image copy: 67% complete... Image copy: 68% complete... Image copy: 70% complete... Image copy: 71% complete... Image copy: 73% complete... Image copy: 75% complete... Image copy: 76% complete...2024-09-21T23:26:46.026+0000 7ff35e8e2640 0 -- 172.21.15.31:0/3121794154 >> [v2:172.21.15.60:6816/1315554966,v1:172.21.15.60:6817/1315554966] conn(0x7ff344008c50 msgr2=0x7ff34400b0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:46.034 INFO:tasks.workunit.client.0.smithi031.stderr: Image copy: 78% complete... Image copy: 79% complete... Image copy: 81% complete... Image copy: 82% complete... Image copy: 84% complete... Image copy: 85% complete... Image copy: 87% complete... Image copy: 89% complete... Image copy: 90% complete... Image copy: 92% complete... Image copy: 93% complete... Image copy: 95% complete... Image copy: 96% complete... Image copy: 98% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2024-09-21T23:26:46.038 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd remove testrbdcopys46970 2024-09-21T23:26:46.080 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:46.079+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:46.111 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:46.111 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:46.111 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:46.111 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:46.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.4244.0:1 2.3 e19/18 2:c4c92e5a:::rbd_trash:head v 19'1, mlcod=0'0)", 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 77.942283506999999, 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039243489999999997, 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.112 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167658+0000", 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2999999999999999e-07 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167667+0000", 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7169999999999997e-06 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167679+0000", 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.276e-05 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167688+0000", 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8380000000000008e-06 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167739+0000", 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0964999999999997e-05 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.114 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.167762+0000", 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2221999999999998e-05 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171515+0000", 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037531119999999999 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:25:28.171582+0000", 2024-09-21T23:26:46.115 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7504999999999998e-05 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5244.0:61 2.1 2:83b64d0f:::rbd_header.13013c9ae692:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.891124+0000", 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.218817324, 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00466922, 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:46.116 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5244", 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3352402684", 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 61 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891124+0000", 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891124+0000", 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891126+0000", 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1100000000000001e-06 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891137+0000", 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0474e-05 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891161+0000", 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4879e-05 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891173+0000", 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1708e-05 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891254+0000", 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0748999999999995e-05 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op must wait for map", 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891261+0000", 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7780000000000002e-06 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.893987+0000", 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002726235 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.894068+0000", 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1336999999999997e-05 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3", 2024-09-21T23:26:46.120 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.894394+0000", 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000325827 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.895449+0000", 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010548510000000001 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.895674+0000", 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000225474 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.895739+0000", 2024-09-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4941e-05 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.895793+0000", 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3857000000000003e-05 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:74 2.6 2:62f7b169:::rbd_data.13013c9ae692.0000000000000003:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.993643+0000", 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.116298644, 2024-09-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044537830000000002, 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 74 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993643+0000", 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993643+0000", 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993643+0000", 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4900000000000005e-07 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993647+0000", 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.014e-06 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.124 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993659+0000", 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1184e-05 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993669+0000", 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0360000000000001e-05 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993745+0000", 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.6540999999999998e-05 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993864+0000", 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000118047 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998096+0000", 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042328879999999998 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:77 2.1 2:9b2c1414:::rbd_data.13013c9ae692.0000000000000006:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.993724+0000", 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.116216885, 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049854790000000001, 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 77 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993724+0000", 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993724+0000", 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993725+0000", 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4700000000000002e-07 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993728+0000", 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0120000000000001e-06 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993736+0000", 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9249999999999995e-06 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993741+0000", 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7039999999999999e-06 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994202+0000", 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00046076199999999998 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994327+0000", 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012460800000000001 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998710+0000", 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004383021 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:84 2.1 2:987fd1fb:::rbd_data.13013c9ae692.000000000000000d:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.997932+0000", 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.112009491, 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050909830000000003, 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:46.131 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 84 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997932+0000", 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997932+0000", 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.132 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997933+0000", 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.235e-06 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997938+0000", 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4020000000000001e-06 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997953+0000", 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4986e-05 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.133 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997962+0000", 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3310000000000004e-06 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998021+0000", 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.889e-05 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998141+0000", 2024-09-21T23:26:46.134 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000120043 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003023+0000", 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048820959999999998 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.3 epoch 57/18 rep_tid 64 entries 57'15 (0'0) error 2:cfc8b22f:::rbd_data.13013c9ae692.000000000000000e:head by client.5259.0:85 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'13)", 2024-09-21T23:26:46.135 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.998638+0000", 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.111302792, 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050258189999999999, 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998638+0000", 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998638+0000", 2024-09-21T23:26:46.136 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998639+0000", 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9999999999999997e-07 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998647+0000", 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9100000000000005e-06 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998660+0000", 2024-09-21T23:26:46.137 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.314e-05 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998669+0000", 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3240000000000003e-06 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998756+0000", 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7517999999999998e-05 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003664+0000", 2024-09-21T23:26:46.138 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049082270000000002 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:87 2.1 2:834d47d4:::rbd_data.13013c9ae692.0000000000000010:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.998828+0000", 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.1111130440000001, 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005133398, 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.139 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 87 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998828+0000", 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998828+0000", 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998829+0000", 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4399999999999998e-07 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998838+0000", 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5590000000000006e-06 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998852+0000", 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4072e-05 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998860+0000", 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7349999999999996e-06 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998916+0000", 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6033999999999999e-05 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998990+0000", 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4908999999999997e-05 2024-09-21T23:26:46.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003962+0000", 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049711449999999997 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:89 2.6 2:7da3782d:::rbd_data.13013c9ae692.0000000000000012:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.001006+0000", 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.108935585, 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0026102679999999998, 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 89 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001006+0000", 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001006+0000", 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001007+0000", 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2399999999999996e-07 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001011+0000", 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.78e-06 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.145 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001025+0000", 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3736e-05 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001033+0000", 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4220000000000003e-06 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001081+0000", 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8399000000000002e-05 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001183+0000", 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000102369 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003616+0000", 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002432638 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.3 epoch 57/18 rep_tid 66 entries 57'16 (0'0) error 2:d61947da:::rbd_data.13013c9ae692.0000000000000014:head by client.5259.0:91 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'13)", 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.003407+0000", 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.1065342279999999, 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028790640000000002, 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003407+0000", 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003407+0000", 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003408+0000", 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7000000000000004e-07 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003416+0000", 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4489999999999999e-06 2024-09-21T23:26:46.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003434+0000", 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7706999999999998e-05 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003450+0000", 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6001000000000001e-05 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003536+0000", 2024-09-21T23:26:46.150 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6296000000000002e-05 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006286+0000", 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027499410000000001 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:97 2.6 2:7a408907:::rbd_data.13013c9ae692.000000000000001a:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.006438+0000", 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.103503328, 2024-09-21T23:26:46.151 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002557765, 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 97 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006438+0000", 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006438+0000", 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006439+0000", 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.28e-06 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006444+0000", 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6589999999999998e-06 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.153 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006459+0000", 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5381000000000001e-05 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006469+0000", 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0142e-05 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006535+0000", 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.5351000000000005e-05 2024-09-21T23:26:46.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006665+0000", 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012996700000000001 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.008996+0000", 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0023309849999999998 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.155 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:121 2.1 2:8afe7a84:::journal.13013c9ae692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.212175+0000", 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.89776671799999996, 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048095250000000003, 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 121 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.156 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212175+0000", 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212175+0000", 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212175+0000", 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8500000000000002e-07 2024-09-21T23:26:46.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212178+0000", 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7350000000000001e-06 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212184+0000", 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6170000000000003e-06 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212188+0000", 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6229999999999999e-06 2024-09-21T23:26:46.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212481+0000", 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000292472 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212567+0000", 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6602999999999997e-05 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3", 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212679+0000", 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111721 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216910+0000", 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042315629999999998 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216934+0000", 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3595999999999999e-05 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216959+0000", 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5014000000000001e-05 2024-09-21T23:26:46.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216984+0000", 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5096000000000002e-05 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:77 2.1 2:8e673ce9:::rbd_data.1286a02efebc.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.305051+0000", 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.80489054400000004, 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036734070000000001, 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 77 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305051+0000", 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.162 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305051+0000", 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305051+0000", 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2900000000000003e-07 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305055+0000", 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7969999999999998e-06 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.163 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305065+0000", 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5920000000000007e-06 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305069+0000", 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6090000000000003e-06 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305128+0000", 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8990000000000003e-05 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305299+0000", 2024-09-21T23:26:46.164 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017046499999999999 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.308724+0000", 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034253249999999999 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:78 2.6 2:7a4ce2c7:::rbd_data.1286a02efebc.0000000000000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.305085+0000", 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.80485593499999997, 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036580480000000001, 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 78 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305085+0000", 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305085+0000", 2024-09-21T23:26:46.166 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305086+0000", 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.89e-07 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305090+0000", 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0640000000000004e-06 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.167 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305099+0000", 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4720000000000001e-06 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305105+0000", 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0530000000000003e-06 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305161+0000", 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5278000000000002e-05 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305294+0000", 2024-09-21T23:26:46.168 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013302 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.308743+0000", 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034497719999999998 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:79 2.6 2:793f91dc:::rbd_data.1286a02efebc.0000000000000008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.305119+0000", 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.80482260500000002, 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0037066730000000002, 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 79 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305119+0000", 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.170 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305119+0000", 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305119+0000", 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7899999999999999e-07 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305122+0000", 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9679999999999998e-06 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.171 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305130+0000", 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5519999999999999e-06 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305136+0000", 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3389999999999998e-06 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305597+0000", 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00046071399999999998 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.172 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305705+0000", 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000108434 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.308825+0000", 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003120187 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.3 epoch 57/18 rep_tid 85 entries 57'22 (0'0) error 2:d2c798c6:::rbd_data.1286a02efebc.0000000000000005:head by client.4946.0:76 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'19)", 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.305757+0000", 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.80418420599999996, 2024-09-21T23:26:46.173 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031988199999999998, 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305757+0000", 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305757+0000", 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305758+0000", 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8500000000000001e-07 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305766+0000", 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8990000000000001e-06 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305791+0000", 2024-09-21T23:26:46.175 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5366000000000001e-05 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305806+0000", 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4980000000000001e-05 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.305940+0000", 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000133685 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.308956+0000", 2024-09-21T23:26:46.176 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0030163049999999999 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:81 2.1 2:8b17895d:::rbd_data.1286a02efebc.000000000000000a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.306422+0000", 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.80351889600000004, 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0027390650000000002, 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:46.177 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 81 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306422+0000", 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306422+0000", 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306423+0000", 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9999999999999996e-07 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306428+0000", 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.391e-06 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306441+0000", 2024-09-21T23:26:46.179 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.324e-05 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306448+0000", 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1470000000000004e-06 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306506+0000", 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8118000000000002e-05 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.180 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306627+0000", 2024-09-21T23:26:46.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000121162 2024-09-21T23:26:46.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.181 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.181 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.309161+0000", 2024-09-21T23:26:46.181 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025341069999999999 2024-09-21T23:26:46.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.181 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:90 2.1 2:873b27ee:::rbd_data.1286a02efebc.0000000000000013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.309941+0000", 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.80000001300000001, 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035439149999999999, 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 90 2024-09-21T23:26:46.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.309941+0000", 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.309941+0000", 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.183 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.309942+0000", 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2800000000000003e-07 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.309944+0000", 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0020000000000001e-06 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.309954+0000", 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9720000000000004e-06 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.309958+0000", 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0010000000000003e-06 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310391+0000", 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00043378200000000002 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310512+0000", 2024-09-21T23:26:46.185 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000120758 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313485+0000", 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029730720000000002 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:98 2.1 2:97617b74:::rbd_data.1286a02efebc.000000000000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.312348+0000", 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.79759323800000004, 2024-09-21T23:26:46.186 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004809598, 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 98 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312348+0000", 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312348+0000", 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312349+0000", 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.78e-07 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312356+0000", 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.691e-06 2024-09-21T23:26:46.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312366+0000", 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0708e-05 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312375+0000", 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7199999999999995e-06 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312539+0000", 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016393700000000001 2024-09-21T23:26:46.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312646+0000", 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106838 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.317158+0000", 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045117259999999998 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:100 2.1 2:8eef717c:::rbd_data.1286a02efebc.000000000000001d:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:46.190 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.312634+0000", 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.79730682100000005, 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046341109999999998, 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 100 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312634+0000", 2024-09-21T23:26:46.191 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312634+0000", 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312635+0000", 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1699999999999997e-07 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312648+0000", 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.225e-05 2024-09-21T23:26:46.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312657+0000", 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0929999999999992e-06 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312661+0000", 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6750000000000001e-06 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313023+0000", 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00036178000000000002 2024-09-21T23:26:46.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313132+0000", 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000109287 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.317269+0000", 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041361089999999998 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.3 epoch 57/18 rep_tid 91 entries 57'27 (0'0) error 2:d00f20ab:::rbd_data.1286a02efebc.000000000000001e:head by client.4946.0:101 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'25)", 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.313624+0000", 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.79631718900000004, 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003330506, 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313624+0000", 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313624+0000", 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313625+0000", 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1399999999999997e-07 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313633+0000", 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1610000000000003e-06 2024-09-21T23:26:46.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313654+0000", 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1447999999999999e-05 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313669+0000", 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4739999999999999e-05 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313750+0000", 2024-09-21T23:26:46.197 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0679999999999993e-05 2024-09-21T23:26:46.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.198 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.198 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.316955+0000", 2024-09-21T23:26:46.198 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032048630000000001 2024-09-21T23:26:46.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.198 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.198 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.198 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:46.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:46.203 INFO:tasks.workunit.client.0.smithi031.stderr:2024-09-21T23:26:46.130+0000 7f2b1861e640 0 -- 172.21.15.31:0/1113259170 >> [v2:172.21.15.31:6808/864917651,v1:172.21.15.31:6809/864917651] conn(0x7f2af4019550 msgr2=0x7f2af40199b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:46.203 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete... Removing image: 100% complete...done. 2024-09-21T23:26:46.203 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdcopy46970 'journal info' //journal/order 25 2024-09-21T23:26:46.203 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdcopy46970 2024-09-21T23:26:46.203 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:46.203 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/order 2024-09-21T23:26:46.203 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=25 2024-09-21T23:26:46.203 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdcopy46970 2024-09-21T23:26:46.203 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/order 2024-09-21T23:26:46.203 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:46.199+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:46.204 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=25 2024-09-21T23:26:46.205 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 25 = 25 2024-09-21T23:26:46.205 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdcopy46970 'journal info' //journal/splay_width 6 2024-09-21T23:26:46.205 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdcopy46970 2024-09-21T23:26:46.205 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:46.205 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/splay_width 2024-09-21T23:26:46.205 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=6 2024-09-21T23:26:46.205 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdcopy46970 2024-09-21T23:26:46.205 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/splay_width 2024-09-21T23:26:46.261 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=6 2024-09-21T23:26:46.261 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 6 = 6 2024-09-21T23:26:46.261 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdcopy46970 'journal info' //journal/object_pool rbd 2024-09-21T23:26:46.261 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdcopy46970 2024-09-21T23:26:46.261 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:46.261 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/object_pool 2024-09-21T23:26:46.261 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=rbd 2024-09-21T23:26:46.261 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdcopy46970 2024-09-21T23:26:46.261 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/object_pool 2024-09-21T23:26:46.300 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:46.299+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:46.315 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=rbd 2024-09-21T23:26:46.315 INFO:tasks.workunit.client.0.smithi031.stderr:+ test rbd = rbd 2024-09-21T23:26:46.315 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd remove testrbdcopy46970 2024-09-21T23:26:46.321 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:46.321 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:46.321 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:46.321 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:46.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:46.397 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete...2024-09-21T23:26:46.396+0000 7f15e5426640 0 -- 172.21.15.31:0/2679851225 >> [v2:172.21.15.60:6800/3490482162,v1:172.21.15.60:6801/3490482162] conn(0x7f15cc008c50 msgr2=0x7f15cc00b110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:46.401 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:46.401+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:46.405 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... 2024-09-21T23:26:46.405+0000 7f15e5c27640 0 -- 172.21.15.31:0/2679851225 >> [v2:172.21.15.60:6824/2320687649,v1:172.21.15.60:6825/2320687649] conn(0x5581a3cac110 msgr2=0x5581a3cae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:46.446 INFO:tasks.workunit.client.0.smithi031.stderr:Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete... Removing image: 100% complete...done. 2024-09-21T23:26:46.449 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:46.449 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:46.449 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:46.449 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:46.449 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:+ set +x 2024-09-21T23:26:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:+ test_rbd_clone 2024-09-21T23:26:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:+ local parent=testrbdclonep46970 2024-09-21T23:26:46.450 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --image-feature layering --size 256 testrbdclonep46970 2024-09-21T23:26:46.459 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:46.498 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd snap create testrbdclonep46970@snap 2024-09-21T23:26:46.502 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:46.502+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:46.603 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:46.602+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:46.609 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:46.609 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:46.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:46.609 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:46.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.609 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.5259.0:65 2.1 e57/18 2:83b64d0f:::rbd_header.13013c9ae692:head v 57'52, mlcod=57'50)", 2024-09-21T23:26:46.609 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.990709+0000", 2024-09-21T23:26:46.609 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.6172885180000001, 2024-09-21T23:26:46.609 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001792344, 2024-09-21T23:26:46.609 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.609 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.990709+0000", 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.990709+0000", 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.610 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.990710+0000", 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9800000000000003e-07 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.990714+0000", 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5170000000000001e-06 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.990723+0000", 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4749999999999999e-06 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.990730+0000", 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7599999999999997e-06 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.990793+0000", 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2768999999999995e-05 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.990814+0000", 2024-09-21T23:26:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1338000000000001e-05 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.992463+0000", 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001649017 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.992501+0000", 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7769999999999999e-05 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.613 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 55 entries 57'54 (0'0) error 2:9b2c1414:::rbd_data.13013c9ae692.0000000000000006:head by client.5259.0:77 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'51)", 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.994604+0000", 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.613392999, 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033897520000000002, 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994604+0000", 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994604+0000", 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994605+0000", 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9999999999999997e-07 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994613+0000", 2024-09-21T23:26:46.615 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0800000000000006e-06 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994631+0000", 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7530999999999999e-05 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994645+0000", 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4372999999999999e-05 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994742+0000", 2024-09-21T23:26:46.616 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7409000000000002e-05 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997994+0000", 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032516590000000001 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 56 entries 57'55 (0'0) error 2:987fd1fb:::rbd_data.13013c9ae692.000000000000000d:head by client.5259.0:84 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'52)", 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.998421+0000", 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.6095761230000001, 2024-09-21T23:26:46.617 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00229803, 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998421+0000", 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998421+0000", 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998422+0000", 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4900000000000004e-07 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998431+0000", 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4849999999999992e-06 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998453+0000", 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2082999999999998e-05 2024-09-21T23:26:46.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998472+0000", 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9490000000000001e-05 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998552+0000", 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0091000000000003e-05 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.000719+0000", 2024-09-21T23:26:46.620 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021669319999999999 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 57 entries 57'56 (0'0) error 2:834d47d4:::rbd_data.13013c9ae692.0000000000000010:head by client.5259.0:87 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'53)", 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.999182+0000", 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.6088155799999999, 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044732239999999996, 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999182+0000", 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999182+0000", 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999182+0000", 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8400000000000004e-07 2024-09-21T23:26:46.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999191+0000", 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3529999999999995e-06 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999213+0000", 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2252e-05 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999229+0000", 2024-09-21T23:26:46.623 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5973000000000001e-05 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999303+0000", 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3640000000000006e-05 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003655+0000", 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043523219999999996 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.5259.0:116 2.1 e57/18 2:9e43db55:::journal_data.2.13013c9ae692.0:head v 57'63, mlcod=57'61)", 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.209475+0000", 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.398522201, 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0019873709999999999, 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209475+0000", 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209475+0000", 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209476+0000", 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5799999999999998e-07 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209482+0000", 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6890000000000001e-06 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.626 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209493+0000", 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1806e-05 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209500+0000", 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8700000000000003e-06 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209553+0000", 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2355000000000003e-05 2024-09-21T23:26:46.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209573+0000", 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0219000000000001e-05 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.211393+0000", 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001819745 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.211462+0000", 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9929000000000001e-05 2024-09-21T23:26:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.5259.0:121 2.1 e57/18 2:8afe7a84:::journal.13013c9ae692:head v 57'65, mlcod=57'62)", 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.212741+0000", 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.395256158, 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040698150000000001, 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212741+0000", 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212741+0000", 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212742+0000", 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4900000000000005e-07 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212758+0000", 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6032000000000002e-05 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212769+0000", 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.095e-05 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212775+0000", 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6500000000000001e-06 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212827+0000", 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2098000000000002e-05 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212838+0000", 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1165000000000001e-05 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216768+0000", 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039302679999999998 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216811+0000", 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2902999999999999e-05 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 74 entries 57'67 (0'0) error 2:8b17895d:::rbd_data.1286a02efebc.000000000000000a:head by client.4946.0:81 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'64)", 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.306823+0000", 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.3011742749999999, 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0018762239999999999, 2024-09-21T23:26:46.633 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306823+0000", 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306823+0000", 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306824+0000", 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9999999999999997e-07 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306832+0000", 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4640000000000006e-06 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306853+0000", 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0687999999999999e-05 2024-09-21T23:26:46.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306868+0000", 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5214e-05 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.306949+0000", 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0542000000000002e-05 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.636 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.308699+0000", 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001750616 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 76 entries 57'69 (0'0) error 2:873b27ee:::rbd_data.1286a02efebc.0000000000000013:head by client.4946.0:90 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'66)", 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.310738+0000", 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.297259462, 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021035989999999998, 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310738+0000", 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310738+0000", 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310738+0000", 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4499999999999997e-07 2024-09-21T23:26:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310745+0000", 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9160000000000004e-06 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310762+0000", 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6240999999999999e-05 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310790+0000", 2024-09-21T23:26:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8316000000000001e-05 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310848+0000", 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8136e-05 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312841+0000", 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0019934449999999999 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 77 entries 57'70 (0'0) error 2:94733709:::rbd_data.1286a02efebc.0000000000000018:head by client.4946.0:95 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'67)", 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.312481+0000", 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.2955163110000001, 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0019222919999999999, 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312481+0000", 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312481+0000", 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312482+0000", 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8299999999999996e-07 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312490+0000", 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.585e-06 2024-09-21T23:26:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312512+0000", 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1379999999999999e-05 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312531+0000", 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9541000000000001e-05 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312630+0000", 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8249000000000001e-05 2024-09-21T23:26:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.314403+0000", 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001773754 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 78 entries 57'71 (0'0) error 2:97617b74:::rbd_data.1286a02efebc.000000000000001b:head by client.4946.0:98 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'67)", 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.312857+0000", 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.2951398840000001, 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040794789999999996, 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312857+0000", 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312857+0000", 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.645 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312858+0000", 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1200000000000003e-07 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312866+0000", 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3000000000000002e-06 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312884+0000", 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7923000000000001e-05 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.646 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312898+0000", 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3322999999999999e-05 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312980+0000", 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2876999999999999e-05 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.316937+0000", 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039564439999999999 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.647 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 79 entries 57'72 (0'0) error 2:8eef717c:::rbd_data.1286a02efebc.000000000000001d:head by client.4946.0:100 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'67)", 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.313337+0000", 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.2946606730000001, 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036166739999999998, 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313337+0000", 2024-09-21T23:26:46.648 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313337+0000", 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313337+0000", 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0999999999999998e-07 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313345+0000", 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8599999999999993e-06 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313366+0000", 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0801e-05 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313382+0000", 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5877000000000001e-05 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313482+0000", 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010050799999999999 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.316953+0000", 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034710180000000002 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.5328.0:52 2.1 e57/18 2:9c628179:::rbd_id.testrbdcopys46970:head v 57'78, mlcod=57'76)", 2024-09-21T23:26:46.651 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.113617+0000", 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.494380502, 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020936610000000001, 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113617+0000", 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113617+0000", 2024-09-21T23:26:46.652 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113618+0000", 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2309999999999999e-06 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113626+0000", 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2819999999999996e-06 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.653 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113643+0000", 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6164e-05 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113650+0000", 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3939999999999998e-06 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113713+0000", 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3495000000000001e-05 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.654 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113737+0000", 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3142999999999998e-05 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.115662+0000", 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0019258840000000001 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.115710+0000", 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8068000000000001e-05 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.655 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 96 entries 57'79 (0'0) error 2:9f98d0d9:::rbd_data.138b5612b65e.0000000000000007:head by client.5373.0:78 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'77)", 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.396762+0000", 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.21123555899999999, 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033279669999999998, 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396762+0000", 2024-09-21T23:26:46.656 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396762+0000", 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396763+0000", 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6449999999999999e-06 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396776+0000", 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.279e-05 2024-09-21T23:26:46.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396814+0000", 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7703999999999997e-05 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396832+0000", 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8117e-05 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396946+0000", 2024-09-21T23:26:46.658 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011387000000000001 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.400090+0000", 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031438410000000001 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 98 entries 57'80 (0'0) error 2:8d7a4cd3:::rbd_data.138b5612b65e.0000000000000009:head by client.5373.0:80 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'77)", 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.397140+0000", 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.21085769900000001, 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055420900000000004, 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397140+0000", 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397140+0000", 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.660 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397140+0000", 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2799999999999996e-07 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397148+0000", 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2730000000000002e-06 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397166+0000", 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7569999999999999e-05 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.661 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397181+0000", 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.453e-05 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397270+0000", 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9159999999999993e-05 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.402682+0000", 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054120289999999996 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.662 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 101 entries 57'81 (0'0) error 2:9679c393:::rbd_data.138b5612b65e.000000000000000b:head by client.5373.0:83 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'77)", 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.398263+0000", 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.20973473200000001, 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044348599999999997, 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398263+0000", 2024-09-21T23:26:46.663 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398263+0000", 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398263+0000", 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4600000000000003e-07 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398275+0000", 2024-09-21T23:26:46.664 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1294000000000001e-05 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398297+0000", 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2517000000000001e-05 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398312+0000", 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5223e-05 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398394+0000", 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1848999999999995e-05 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.402697+0000", 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004303131 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 105 entries 57'82 (0'0) error 2:979d07b6:::rbd_data.138b5612b65e.000000000000001c:head by client.5373.0:100 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'80)", 2024-09-21T23:26:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.406175+0000", 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.20182209300000001, 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002193515, 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406175+0000", 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.667 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406175+0000", 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406179+0000", 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0419999999999996e-06 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406190+0000", 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0848e-05 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.668 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406217+0000", 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7067e-05 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406234+0000", 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7011999999999999e-05 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406310+0000", 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5791000000000007e-05 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.669 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.408369+0000", 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020587549999999998 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 107 entries 57'84 (0'0) error 2:8d3177d6:::rbd_data.138b5612b65e.0000000000000029:head by client.5373.0:113 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'81)", 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.411565+0000", 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.19643264499999999, 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040547370000000001, 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.670 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411565+0000", 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411565+0000", 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.671 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411565+0000", 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2099999999999996e-07 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411573+0000", 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0020000000000006e-06 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411590+0000", 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6986999999999998e-05 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411603+0000", 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3169999999999999e-05 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411705+0000", 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010180500000000001 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415619+0000", 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039141519999999997 2024-09-21T23:26:46.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 109 entries 57'85 (0'0) error 2:99397b72:::rbd_data.138b5612b65e.000000000000002f:head by client.5373.0:119 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'82)", 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.414176+0000", 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.19382091200000001, 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038963309999999998, 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.674 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414176+0000", 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414176+0000", 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414177+0000", 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8800000000000002e-07 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414186+0000", 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2129999999999997e-06 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414210+0000", 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3988000000000001e-05 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414227+0000", 2024-09-21T23:26:46.676 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6714999999999998e-05 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414325+0000", 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8095000000000002e-05 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.418073+0000", 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003747532 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 110 entries 57'86 (0'0) error 2:94fb10ec:::rbd_data.138b5612b65e.0000000000000030:head by client.5373.0:120 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'82)", 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.414564+0000", 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.19343347999999999, 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035261559999999999, 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414564+0000", 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414564+0000", 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414564+0000", 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2099999999999996e-07 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414572+0000", 2024-09-21T23:26:46.679 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7060000000000004e-06 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414593+0000", 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0902000000000001e-05 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414608+0000", 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.446e-05 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.680 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414694+0000", 2024-09-21T23:26:46.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6805999999999995e-05 2024-09-21T23:26:46.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.681 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.681 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.418090+0000", 2024-09-21T23:26:46.681 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033956609999999999 2024-09-21T23:26:46.681 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.681 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.681 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.682 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 111 entries 57'87 (0'0) error 2:8b7e6962:::rbd_data.138b5612b65e.0000000000000032:head by client.5373.0:122 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'82)", 2024-09-21T23:26:46.682 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.415021+0000", 2024-09-21T23:26:46.682 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.192976285, 2024-09-21T23:26:46.682 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003083026, 2024-09-21T23:26:46.682 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:46.682 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:46.682 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415021+0000", 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415021+0000", 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.683 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415022+0000", 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0100000000000005e-07 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415029+0000", 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4599999999999997e-06 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415047+0000", 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7419e-05 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415059+0000", 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2211e-05 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415113+0000", 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4506999999999998e-05 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.418104+0000", 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029908280000000001 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.685 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.686 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.686 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:46.686 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:46.686 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:46.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:46.704 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:46.704+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:46.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T23:26:46.795 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:46.796 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:46.796 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:46.796 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:46.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:46.805 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:46.804+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:46.866 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:46.867 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:46.883 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:46.893 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T23:26:46.893 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T23:26:46.893 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T23:26:46.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T23:26:46.902 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:46.902 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:46.905 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:46.904+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:46.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T23:26:46.909 INFO:tasks.workunit.client.0.smithi031.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T23:26:46.912 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:46.912 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:46.912 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:46.912 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:46.912 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:46.913 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd snap protect testrbdclonep46970@snap 2024-09-21T23:26:46.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:46.921 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:46.939 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:46.939 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:46.959 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:46.959 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:46.969 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdclone46970 2024-09-21T23:26:46.969 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd clone --image-feature layering --image-feature exclusive-lock --image-feature journaling --journal-pool rbd --journal-object-size 20M --journal-splay-width 6 testrbdclonep46970@snap testrbdclone46970 2024-09-21T23:26:46.979 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:46.980 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:46.997 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:47.005 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:47.004+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:47.070 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:47.070 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:47.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:47.120 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:47.119+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:26:47.203 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 58, 2024-09-21T23:26:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: "fsid": "d51cc561-ddb8-41ef-b61c-7c6d698e4dd0", 2024-09-21T23:26:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: "created": "2024-09-21T23:25:09.056809+0000", 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "modified": "2024-09-21T23:26:46.897001+0000", 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "last_up_change": "2024-09-21T23:25:20.126996+0000", 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "last_in_change": "2024-09-21T23:25:11.226810+0000", 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_num": 5799936, 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_set": [ 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "pglog_hardlimit", 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "purged_snapdirs", 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_deletes", 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "sortbitwise" 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_version": 5, 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T23:26:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_snapshot": "", 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_max": 2, 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd": 8, 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "require_min_compat_client": "luminous", 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "min_compat_client": "luminous", 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "require_osd_release": "squid", 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_crimson": false, 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "pools": [ 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 1, 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": ".mgr", 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:19.818212+0000", 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-09-21T23:26:47.205 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool", 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 1, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 1, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 1, 2024-09-21T23:26:47.206 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 1, 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 1, 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "17", 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:47.207 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 0, 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 0, 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:47.209 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "options": { 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_max": 32, 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_min": 1 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": {} 2024-09-21T23:26:47.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 6.9000000953674316, 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 6.9000000953674316, 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 0.28999999165534973, 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 2, 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": "rbd", 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "create_time": "2024-09-21T23:25:26.968096+0000", 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 8193, 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "type": 1, 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 2, 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": 1, 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_rule": 0, 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_count": 0, 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_target": 0, 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "is_stretch_pool": false, 2024-09-21T23:26:47.212 INFO:teuthology.orchestra.run.smithi031.stdout: "object_hash": 2, 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscale_mode": "off", 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": 8, 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num": 8, 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_placement_num_target": 8, 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_target": 8, 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num_pending": 8, 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "last_pg_merge_meta": { 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "source_pgid": "0.0", 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "ready_epoch": 0, 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_started": 0, 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "last_epoch_clean": 0, 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "source_version": "0'0", 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "target_version": "0'0" 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.213 INFO:teuthology.orchestra.run.smithi031.stdout: "last_change": "58", 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend": "0", 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "auid": 0, 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_mode": "selfmanaged", 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_seq": 4, 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_epoch": 58, 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_snaps": [], 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps": "[]", 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_bytes": 0, 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "quota_max_objects": 0, 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "tiers": [], 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_of": -1, 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "read_tier": -1, 2024-09-21T23:26:47.214 INFO:teuthology.orchestra.run.smithi031.stdout: "write_tier": -1, 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_mode": "none", 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_bytes": 0, 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "target_max_objects": 0, 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_flush_age": 0, 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "cache_min_evict_age": 0, 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profile": "", 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_params": { 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "none" 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_period": 0, 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_count": 0, 2024-09-21T23:26:47.215 INFO:teuthology.orchestra.run.smithi031.stdout: "use_gmt_hitset": true, 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "min_read_recency_for_promote": 0, 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "min_write_recency_for_promote": 0, 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "hit_set_search_last_n": 0, 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "grade_table": [], 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "stripe_width": 0, 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "expected_num_objects": 0, 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "fast_read": false, 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "options": {}, 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "application_metadata": { 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd": {} 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.216 INFO:teuthology.orchestra.run.smithi031.stdout: "read_balance": { 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: "score_type": "Fair distribution", 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: "score_acting": 1.75, 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: "score_stable": 1.75, 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: "optimal_score": 1.1399999856948853, 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_acting": 2, 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: "raw_score_stable": 2, 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity_weighted": 0.875, 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity": 1, 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:47.217 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af", 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 0, 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 35, 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6816", 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6817", 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6818", 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6819", 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6822", 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6823", 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6820", 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6821", 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2339966024 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6817/2339966024", 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6819/2339966024", 2024-09-21T23:26:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6823/2339966024", 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6821/2339966024", 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "88ee06a3-9a6a-41e8-9327-a7a61304c03d", 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 49, 2024-09-21T23:26:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6824", 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6825", 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6826", 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6827", 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6830", 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.227 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6831", 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6828", 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6829", 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2889649822 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6825/2889649822", 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6827/2889649822", 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6831/2889649822", 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6829/2889649822", 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:47.229 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05", 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:47.230 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 42, 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6808", 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6809", 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6810", 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:47.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6811", 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6814", 2024-09-21T23:26:47.233 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6815", 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.234 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6812", 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6813", 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 864917651 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6809/864917651", 2024-09-21T23:26:47.235 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6811/864917651", 2024-09-21T23:26:47.236 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6815/864917651", 2024-09-21T23:26:47.236 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6813/864917651", 2024-09-21T23:26:47.236 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:47.236 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:47.236 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:47.236 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.236 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:47.236 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "562e8c30-0622-4cb3-b5d1-f15b42a444bc", 2024-09-21T23:26:47.236 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 48, 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.237 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.238 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-09-21T23:26:47.238 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:47.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.238 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.238 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-09-21T23:26:47.238 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:47.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.238 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.238 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:47.239 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.239 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.239 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6802", 2024-09-21T23:26:47.239 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:47.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.239 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.239 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6803", 2024-09-21T23:26:47.239 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:47.239 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6806", 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.240 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6807", 2024-09-21T23:26:47.241 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:47.241 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.241 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.241 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:47.241 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.241 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.241 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6804", 2024-09-21T23:26:47.241 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:47.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.242 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6805", 2024-09-21T23:26:47.242 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1713632981 2024-09-21T23:26:47.242 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.242 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.242 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.31:6801/1713632981", 2024-09-21T23:26:47.242 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.31:6803/1713632981", 2024-09-21T23:26:47.242 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.31:6807/1713632981", 2024-09-21T23:26:47.242 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.31:6805/1713632981", 2024-09-21T23:26:47.243 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:47.243 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:47.243 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:47.243 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.243 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:47.243 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7030ff68-4556-4ad5-abc9-be758f1caa85", 2024-09-21T23:26:47.243 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:47.243 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 0, 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.244 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.245 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6808", 2024-09-21T23:26:47.245 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:47.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.245 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6809", 2024-09-21T23:26:47.245 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:47.245 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.245 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.245 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:47.246 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.246 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.246 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6810", 2024-09-21T23:26:47.246 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:47.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.246 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.246 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6811", 2024-09-21T23:26:47.246 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:47.246 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6814", 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.247 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6815", 2024-09-21T23:26:47.248 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:47.248 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.248 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.248 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:47.248 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.248 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.248 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6812", 2024-09-21T23:26:47.248 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:47.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6813", 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2660753894 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6809/2660753894", 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6811/2660753894", 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6815/2660753894", 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6813/2660753894", 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:47.249 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:47.250 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:47.250 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.250 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:47.250 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "7169a163-f067-4385-b734-f58a3ba9e89e", 2024-09-21T23:26:47.250 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:47.250 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:47.250 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:47.250 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:47.250 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:47.251 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:47.251 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 15, 2024-09-21T23:26:47.251 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:47.251 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:47.251 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:47.251 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:47.251 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.251 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6824", 2024-09-21T23:26:47.251 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6825", 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.252 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6826", 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6827", 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6830", 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6831", 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6828", 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6829", 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2320687649 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6825/2320687649", 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6827/2320687649", 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6831/2320687649", 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6829/2320687649", 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:47.256 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "e511295d-2177-4dcb-a74b-b16b92534df6", 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 13, 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6800", 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6801", 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:47.258 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6802", 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6803", 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6806", 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6807", 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:47.260 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6804", 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.261 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6805", 2024-09-21T23:26:47.262 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3490482162 2024-09-21T23:26:47.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.262 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.262 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6801/3490482162", 2024-09-21T23:26:47.262 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6803/3490482162", 2024-09-21T23:26:47.262 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3490482162", 2024-09-21T23:26:47.262 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6805/3490482162", 2024-09-21T23:26:47.262 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:47.262 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:47.262 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: "uuid": "544f8a91-ce82-4855-85eb-ea814eb3d058", 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: "up": 1, 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: "in": 1, 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: "weight": 1, 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_affinity": 1, 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_begin": 0, 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: "last_clean_end": 0, 2024-09-21T23:26:47.263 INFO:teuthology.orchestra.run.smithi031.stdout: "up_from": 14, 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: "up_thru": 18, 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: "down_at": 0, 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: "lost_at": 0, 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addrs": { 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6816", 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6817", 2024-09-21T23:26:47.264 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addrs": { 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6818", 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6819", 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addrs": { 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6822", 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:47.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6823", 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addrs": { 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6820", 2024-09-21T23:26:47.267 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.60:6821", 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1315554966 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: "public_addr": "172.21.15.60:6817/1315554966", 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_addr": "172.21.15.60:6819/1315554966", 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_back_addr": "172.21.15.60:6823/1315554966", 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: "heartbeat_front_addr": "172.21.15.60:6821/1315554966", 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: "state": [ 2024-09-21T23:26:47.268 INFO:teuthology.orchestra.run.smithi031.stdout: "exists", 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: "up" 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_xinfo": [ 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 0, 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:47.269 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.514872+0000", 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 1, 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.912129+0000", 2024-09-21T23:26:47.270 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 2, 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.745154+0000", 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 3, 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:16.758871+0000", 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 4, 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:47.272 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.287462+0000", 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 5, 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:47.273 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:18.031636+0000", 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 6, 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.055855+0000", 2024-09-21T23:26:47.274 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": 7, 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: "down_stamp": "0.000000", 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_probability": 0, 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: "laggy_interval": 0, 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: "features": 4540701547738038271, 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: "old_weight": 0, 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: "last_purged_snaps_scrub": "2024-09-21T23:25:17.524609+0000", 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: "dead_epoch": 0 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:47.275 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap": [ 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "1.0", 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: 0, 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: 1 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: "pgid": "2.6", 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: "osds": [ 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: 2, 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: 4 2024-09-21T23:26:47.276 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_items": [], 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_upmap_primaries": [], 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_temp": [], 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: "primary_temp": [], 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklist": {}, 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: "range_blocklist": {}, 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: "erasure_code_profiles": { 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: "default": { 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-failure-domain": "osd", 2024-09-21T23:26:47.277 INFO:teuthology.orchestra.run.smithi031.stdout: "k": "2", 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: "m": "1", 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: "plugin": "jerasure", 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: "technique": "reed_sol_van" 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: "removed_snaps_queue": [ 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: "pool": 2, 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: "snaps": [ 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: "begin": 3, 2024-09-21T23:26:47.278 INFO:teuthology.orchestra.run.smithi031.stdout: "length": 1 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: "new_removed_snaps": [], 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: "new_purged_snaps": [], 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_node_flags": {}, 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: "device_class_flags": {}, 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode": { 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_enabled": false, 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_bucket_count": 0, 2024-09-21T23:26:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: "degraded_stretch_mode": 0, 2024-09-21T23:26:47.280 INFO:teuthology.orchestra.run.smithi031.stdout: "recovering_stretch_mode": 0, 2024-09-21T23:26:47.280 INFO:teuthology.orchestra.run.smithi031.stdout: "stretch_mode_bucket": 0 2024-09-21T23:26:47.280 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:47.280 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:47.282 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:47.282 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:47.283 INFO:tasks.thrashosds.thrasher:j is {'epoch': 58, 'fsid': 'd51cc561-ddb8-41ef-b61c-7c6d698e4dd0', 'created': '2024-09-21T23:25:09.056809+0000', 'modified': '2024-09-21T23:26:46.897001+0000', 'last_up_change': '2024-09-21T23:25:20.126996+0000', 'last_in_change': '2024-09-21T23:25:11.226810+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T23:25:19.818212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T23:25:26.968096+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 58, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6816', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6817', 'nonce': 2339966024}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6818', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6819', 'nonce': 2339966024}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6822', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6823', 'nonce': 2339966024}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6820', 'nonce': 2339966024}, {'type': 'v1', 'addr': '172.21.15.31:6821', 'nonce': 2339966024}]}, 'public_addr': '172.21.15.31:6817/2339966024', 'cluster_addr': '172.21.15.31:6819/2339966024', 'heartbeat_back_addr': '172.21.15.31:6823/2339966024', 'heartbeat_front_addr': '172.21.15.31:6821/2339966024', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '88ee06a3-9a6a-41e8-9327-a7a61304c03d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 49, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6824', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6825', 'nonce': 2889649822}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6826', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6827', 'nonce': 2889649822}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6830', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6831', 'nonce': 2889649822}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6828', 'nonce': 2889649822}, {'type': 'v1', 'addr': '172.21.15.31:6829', 'nonce': 2889649822}]}, 'public_addr': '172.21.15.31:6825/2889649822', 'cluster_addr': '172.21.15.31:6827/2889649822', 'heartbeat_back_addr': '172.21.15.31:6831/2889649822', 'heartbeat_front_addr': '172.21.15.31:6829/2889649822', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 42, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6808', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6809', 'nonce': 864917651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6810', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6811', 'nonce': 864917651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6814', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6815', 'nonce': 864917651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6812', 'nonce': 864917651}, {'type': 'v1', 'addr': '172.21.15.31:6813', 'nonce': 864917651}]}, 'public_addr': '172.21.15.31:6809/864917651', 'cluster_addr': '172.21.15.31:6811/864917651', 'heartbeat_back_addr': '172.21.15.31:6815/864917651', 'heartbeat_front_addr': '172.21.15.31:6813/864917651', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '562e8c30-0622-4cb3-b5d1-f15b42a444bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6800', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6801', 'nonce': 1713632981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6802', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6803', 'nonce': 1713632981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6806', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6807', 'nonce': 1713632981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6804', 'nonce': 1713632981}, {'type': 'v1', 'addr': '172.21.15.31:6805', 'nonce': 1713632981}]}, 'public_addr': '172.21.15.31:6801/1713632981', 'cluster_addr': '172.21.15.31:6803/1713632981', 'heartbeat_back_addr': '172.21.15.31:6807/1713632981', 'heartbeat_front_addr': '172.21.15.31:6805/1713632981', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7030ff68-4556-4ad5-abc9-be758f1caa85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6809', 'nonce': 2660753894}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6811', 'nonce': 2660753894}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6815', 'nonce': 2660753894}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 2660753894}, {'type': 'v1', 'addr': '172.21.15.60:6813', 'nonce': 2660753894}]}, 'public_addr': '172.21.15.60:6809/2660753894', 'cluster_addr': '172.21.15.60:6811/2660753894', 'heartbeat_back_addr': '172.21.15.60:6815/2660753894', 'heartbeat_front_addr': '172.21.15.60:6813/2660753894', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7169a163-f067-4385-b734-f58a3ba9e89e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6824', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6825', 'nonce': 2320687649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6826', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6827', 'nonce': 2320687649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6830', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6831', 'nonce': 2320687649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6828', 'nonce': 2320687649}, {'type': 'v1', 'addr': '172.21.15.60:6829', 'nonce': 2320687649}]}, 'public_addr': '172.21.15.60:6825/2320687649', 'cluster_addr': '172.21.15.60:6827/2320687649', 'heartbeat_back_addr': '172.21.15.60:6831/2320687649', 'heartbeat_front_addr': '172.21.15.60:6829/2320687649', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e511295d-2177-4dcb-a74b-b16b92534df6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6801', 'nonce': 3490482162}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6803', 'nonce': 3490482162}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6807', 'nonce': 3490482162}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3490482162}, {'type': 'v1', 'addr': '172.21.15.60:6805', 'nonce': 3490482162}]}, 'public_addr': '172.21.15.60:6801/3490482162', 'cluster_addr': '172.21.15.60:6803/3490482162', 'heartbeat_back_addr': '172.21.15.60:6807/3490482162', 'heartbeat_front_addr': '172.21.15.60:6805/3490482162', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '544f8a91-ce82-4855-85eb-ea814eb3d058', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6816', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6817', 'nonce': 1315554966}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6818', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6819', 'nonce': 1315554966}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6822', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6823', 'nonce': 1315554966}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6820', 'nonce': 1315554966}, {'type': 'v1', 'addr': '172.21.15.60:6821', 'nonce': 1315554966}]}, 'public_addr': '172.21.15.60:6817/1315554966', 'cluster_addr': '172.21.15.60:6819/1315554966', 'heartbeat_back_addr': '172.21.15.60:6823/1315554966', 'heartbeat_front_addr': '172.21.15.60:6821/1315554966', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.514872+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.912129+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.745154+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:16.758871+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.287462+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:18.031636+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.055855+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T23:25:17.524609+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [0, 1]}, {'pgid': '2.6', 'osds': [2, 4]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 3, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T23:26:47.283 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-21T23:26:47.284 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:47.284 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:47.281+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:47.284 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdclone46970 'journal info' //journal/order 25 2024-09-21T23:26:47.284 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdclone46970 2024-09-21T23:26:47.285 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:47.285 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/order 2024-09-21T23:26:47.285 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=25 2024-09-21T23:26:47.285 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdclone46970 2024-09-21T23:26:47.285 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/order 2024-09-21T23:26:47.324 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=25 2024-09-21T23:26:47.325 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 25 = 25 2024-09-21T23:26:47.325 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdclone46970 'journal info' //journal/splay_width 6 2024-09-21T23:26:47.325 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdclone46970 2024-09-21T23:26:47.325 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:47.325 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/splay_width 2024-09-21T23:26:47.325 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=6 2024-09-21T23:26:47.325 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdclone46970 2024-09-21T23:26:47.325 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/splay_width 2024-09-21T23:26:47.382 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:47.381+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:47.384 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=6 2024-09-21T23:26:47.384 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 6 = 6 2024-09-21T23:26:47.384 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdclone46970 'journal info' //journal/object_pool rbd 2024-09-21T23:26:47.384 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdclone46970 2024-09-21T23:26:47.384 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:47.384 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/object_pool 2024-09-21T23:26:47.384 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=rbd 2024-09-21T23:26:47.384 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdclone46970 2024-09-21T23:26:47.384 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/object_pool 2024-09-21T23:26:47.431 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:47.431 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:47.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:47.443 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=rbd 2024-09-21T23:26:47.443 INFO:tasks.workunit.client.0.smithi031.stderr:+ test rbd = rbd 2024-09-21T23:26:47.443 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd remove testrbdclone46970 2024-09-21T23:26:47.485 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:47.484+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:47.518 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete...2024-09-21T23:26:47.518+0000 7fcfff41f640 0 -- 172.21.15.31:0/2966640229 >> [v2:172.21.15.31:6824/2889649822,v1:172.21.15.31:6825/2889649822] conn(0x55de4af53df0 msgr2=0x55de4af561e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:47.524 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete...2024-09-21T23:26:47.523+0000 7fcfff41f640 0 -- 172.21.15.31:0/2966640229 >> [v2:172.21.15.31:6808/864917651,v1:172.21.15.31:6809/864917651] conn(0x7fcfdc008c50 msgr2=0x7fcfdc00b110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:47.556 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:47.556 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:47.557 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete... Removing image: 100% complete...done. 2024-09-21T23:26:47.560 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd snap unprotect testrbdclonep46970@snap 2024-09-21T23:26:47.565 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:47.606 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:47.605+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:47.611 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd snap purge testrbdclonep46970 2024-09-21T23:26:47.686 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:47.685+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:47.713 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:47.713 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:47.722 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:47.786 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:47.786+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:47.829 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:47.829 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:47.837 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:47.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:47.887+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:47.912 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is set 2024-09-21T23:26:47.915 INFO:tasks.workunit.client.0.smithi031.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-21T23:26:47.926 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd remove testrbdclonep46970 2024-09-21T23:26:47.945 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:47.945 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:47.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:47.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:47.988+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:48.011 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete...2024-09-21T23:26:48.011+0000 7fac2e76d640 0 -- 172.21.15.31:0/1525254893 >> [v2:172.21.15.60:6824/2320687649,v1:172.21.15.60:6825/2320687649] conn(0x56061b2723a0 msgr2=0x56061b274830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:48.021 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete...2024-09-21T23:26:48.021+0000 7fac2df6c640 0 -- 172.21.15.31:0/1525254893 >> [v2:172.21.15.31:6808/864917651,v1:172.21.15.31:6809/864917651] conn(0x7fac080012a0 msgr2=0x7fac08001750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:48.037 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete... Removing image: 100% complete...done. 2024-09-21T23:26:48.040 INFO:tasks.workunit.client.0.smithi031.stderr:+ set +x 2024-09-21T23:26:48.041 INFO:tasks.workunit.client.0.smithi031.stderr:+ test_rbd_import 2024-09-21T23:26:48.041 INFO:tasks.workunit.client.0.smithi031.stderr:+ local src=testrbdimports46970 2024-09-21T23:26:48.041 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 256 testrbdimports46970 2024-09-21T23:26:48.070 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:48.070 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:48.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:48.088 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd export testrbdimports46970 /tmp/rbd_journal46970/testrbdimports46970.export 2024-09-21T23:26:48.124 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:48.123+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:48.176 INFO:tasks.workunit.client.0.smithi031.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-09-21T23:26:48.181 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd remove testrbdimports46970 2024-09-21T23:26:48.188 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:48.189 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:48.189 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:48.188+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:48.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:48.266 INFO:tasks.workunit.client.0.smithi031.stderr:2024-09-21T23:26:48.265+0000 7f5150a1d640 0 -- 172.21.15.31:0/3583012143 >> [v2:172.21.15.60:6816/1315554966,v1:172.21.15.60:6817/1315554966] conn(0x7f5134008c50 msgr2=0x7f513400b110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:48.278 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2024-09-21T23:26:48.278+0000 7f515921e640 0 -- 172.21.15.31:0/3583012143 >> [v2:172.21.15.60:6824/2320687649,v1:172.21.15.60:6825/2320687649] conn(0x5614a3cb0ed0 msgr2=0x5614a3cb3360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:48.278 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 100% complete...done. 2024-09-21T23:26:48.281 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdimport46970 2024-09-21T23:26:48.282 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd import --image-feature exclusive-lock --image-feature journaling --journal-pool rbd --journal-object-size 20M --journal-splay-width 6 /tmp/rbd_journal46970/testrbdimports46970.export testrbdimport46970 2024-09-21T23:26:48.290 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:48.289+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:48.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T23:26:48.390 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:48.390+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:48.478 INFO:tasks.workunit.client.0.smithi031.stderr: Importing image: 1% complete... Importing image: 3% complete... Importing image: 4% complete... Importing image: 6% complete... Importing image: 7% complete... Importing image: 9% complete... Importing image: 10% complete... Importing image: 12% complete... Importing image: 14% complete... Importing image: 15% complete... Importing image: 17% complete... Importing image: 18% complete... Importing image: 20% complete... Importing image: 21% complete... Importing image: 23% complete... Importing image: 25% complete... Importing image: 26% complete... Importing image: 28% complete... Importing image: 29% complete... Importing image: 31% complete... Importing image: 32% complete... Importing image: 34% complete... Importing image: 35% complete... Importing image: 37% complete... Importing image: 39% complete... Importing image: 40% complete... Importing image: 42% complete... Importing image: 43% complete... Importing image: 45% complete... Importing image: 46% complete... Importing image: 48% complete... Importing image: 50% complete... Importing image: 51% complete... Importing image: 53% complete... Importing image: 54% complete... Importing image: 56% complete... Importing image: 57% complete... Importing image: 59% complete... Importing image: 60% complete... Importing image: 62% complete... Importing image: 64% complete... Importing image: 65% complete... Importing image: 67% complete... Importing image: 68% complete... Importing image: 70% complete... Importing image: 71% complete... Importing image: 73% complete... Importing image: 75% complete... Importing image: 76% complete... Importing image: 78% complete... Importing image: 79% complete... Importing image: 81% complete... Importing image: 82% complete... Importing image: 84% complete... Importing image: 85% complete... Importing image: 87% complete... Importing image: 89% complete... Importing image: 90% complete... Importing image: 92% complete... Importing image: 93% complete... Importing image: 95% complete... Importing image: 96% complete... Importing image: 98% complete... Importing image: 100% complete...done. 2024-09-21T23:26:48.481 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdimport46970 'journal info' //journal/order 25 2024-09-21T23:26:48.481 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdimport46970 2024-09-21T23:26:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/order 2024-09-21T23:26:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=25 2024-09-21T23:26:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdimport46970 2024-09-21T23:26:48.482 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/order 2024-09-21T23:26:48.482 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:48.483 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:48.491 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:48.490+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:48.492 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T23:26:48.536 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=25 2024-09-21T23:26:48.536 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 25 = 25 2024-09-21T23:26:48.536 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdimport46970 'journal info' //journal/splay_width 6 2024-09-21T23:26:48.537 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdimport46970 2024-09-21T23:26:48.537 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:48.537 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/splay_width 2024-09-21T23:26:48.537 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=6 2024-09-21T23:26:48.537 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdimport46970 2024-09-21T23:26:48.537 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/splay_width 2024-09-21T23:26:48.591 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:48.591+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:48.641 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:48.641 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:48.649 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T23:26:48.692 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-21T23:26:48.692+0000 7fc46ded8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33555) UID: 0 2024-09-21T23:26:48.792 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=6 2024-09-21T23:26:48.792 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 6 = 6 2024-09-21T23:26:48.792 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdimport46970 'journal info' //journal/object_pool rbd 2024-09-21T23:26:48.792 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdimport46970 2024-09-21T23:26:48.792 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:48.792 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/object_pool 2024-09-21T23:26:48.792 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=rbd 2024-09-21T23:26:48.792 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdimport46970 2024-09-21T23:26:48.792 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/object_pool 2024-09-21T23:26:48.793 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:48.793+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:48.798 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:48.798 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:48.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T23:26:48.845 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=rbd 2024-09-21T23:26:48.845 INFO:tasks.workunit.client.0.smithi031.stderr:+ test rbd = rbd 2024-09-21T23:26:48.845 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd remove testrbdimport46970 2024-09-21T23:26:48.893 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:48.892+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:48.923 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete...2024-09-21T23:26:48.922+0000 7fca52a4c640 0 -- 172.21.15.31:0/4207132716 >> [v2:172.21.15.31:6808/864917651,v1:172.21.15.31:6809/864917651] conn(0x7fca34004f30 msgr2=0x7fca34007320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:48.929 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete...2024-09-21T23:26:48.928+0000 7fca544af640 0 -- 172.21.15.31:0/4207132716 >> [v2:172.21.15.60:6816/1315554966,v1:172.21.15.60:6817/1315554966] conn(0x7fca300156e0 msgr2=0x7fca300097d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T23:26:48.963 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete... Removing image: 100% complete...done. 2024-09-21T23:26:48.966 INFO:tasks.workunit.client.0.smithi031.stderr:+ set +x 2024-09-21T23:26:48.966 INFO:tasks.workunit.client.0.smithi031.stderr:+ test_rbd_feature 2024-09-21T23:26:48.966 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdfeature46970 2024-09-21T23:26:48.966 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --image-feature exclusive-lock --size 256 testrbdfeature46970 2024-09-21T23:26:48.968 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:48.968 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:48.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T23:26:49.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T23:26:49.016 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd feature enable testrbdfeature46970 journaling --journal-pool rbd --journal-object-size 20M --journal-splay-width 6 2024-09-21T23:26:49.017 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-21T23:26:49.016+0000 7fc551913640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33560) UID: 0 2024-09-21T23:26:49.086 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdfeature46970 'journal info' //journal/order 25 2024-09-21T23:26:49.086 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdfeature46970 2024-09-21T23:26:49.087 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:49.087 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/order 2024-09-21T23:26:49.087 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=25 2024-09-21T23:26:49.087 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdfeature46970 2024-09-21T23:26:49.087 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/order 2024-09-21T23:26:49.094 INFO:teuthology.orchestra.run.smithi031.stdout:ERROR: (22) Invalid argument 2024-09-21T23:26:49.094 INFO:teuthology.orchestra.run.smithi031.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T23:26:49.095 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:49.095+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:49.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T23:26:49.144 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=25 2024-09-21T23:26:49.144 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 25 = 25 2024-09-21T23:26:49.144 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdfeature46970 'journal info' //journal/splay_width 6 2024-09-21T23:26:49.144 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdfeature46970 2024-09-21T23:26:49.144 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:49.144 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/splay_width 2024-09-21T23:26:49.144 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=6 2024-09-21T23:26:49.145 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdfeature46970 2024-09-21T23:26:49.145 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/splay_width 2024-09-21T23:26:49.151 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0: {} 2024-09-21T23:26:49.151 INFO:teuthology.orchestra.run.smithi031.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T23:26:49.170 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1: {} 2024-09-21T23:26:49.171 INFO:teuthology.orchestra.run.smithi031.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T23:26:49.191 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2: {} 2024-09-21T23:26:49.191 INFO:teuthology.orchestra.run.smithi031.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T23:26:49.196 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:49.195+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:49.202 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=6 2024-09-21T23:26:49.202 INFO:tasks.workunit.client.0.smithi031.stderr:+ test 6 = 6 2024-09-21T23:26:49.202 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd_assert_eq testrbdfeature46970 'journal info' //journal/object_pool rbd 2024-09-21T23:26:49.202 INFO:tasks.workunit.client.0.smithi031.stderr:+ local image=testrbdfeature46970 2024-09-21T23:26:49.202 INFO:tasks.workunit.client.0.smithi031.stderr:+ local 'cmd=journal info' 2024-09-21T23:26:49.202 INFO:tasks.workunit.client.0.smithi031.stderr:+ local param=//journal/object_pool 2024-09-21T23:26:49.202 INFO:tasks.workunit.client.0.smithi031.stderr:+ local expected_val=rbd 2024-09-21T23:26:49.203 INFO:tasks.workunit.client.0.smithi031.stderr:++ rbd --format xml journal info --image testrbdfeature46970 2024-09-21T23:26:49.203 INFO:tasks.workunit.client.0.smithi031.stderr:++ xmlstarlet sel -t -v //journal/object_pool 2024-09-21T23:26:49.210 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3: {} 2024-09-21T23:26:49.210 INFO:teuthology.orchestra.run.smithi031.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T23:26:49.227 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4: {} 2024-09-21T23:26:49.228 INFO:teuthology.orchestra.run.smithi031.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T23:26:49.247 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5: {} 2024-09-21T23:26:49.247 INFO:teuthology.orchestra.run.smithi031.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T23:26:49.260 INFO:tasks.workunit.client.0.smithi031.stderr:+ local val=rbd 2024-09-21T23:26:49.260 INFO:tasks.workunit.client.0.smithi031.stderr:+ test rbd = rbd 2024-09-21T23:26:49.260 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd remove testrbdfeature46970 2024-09-21T23:26:49.263 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:49.263 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:49.263 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:49.263 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:49.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.263 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:73 2.4 2:28a01f35:::rbd_data.13013c9ae692.0000000000000002:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.993648+0000", 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2677199750000003, 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048601310000000002, 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 73 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993648+0000", 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993648+0000", 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993649+0000", 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.728e-06 2024-09-21T23:26:49.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993658+0000", 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5010000000000004e-06 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993677+0000", 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9122000000000001e-05 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993689+0000", 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1752e-05 2024-09-21T23:26:49.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993770+0000", 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1304999999999995e-05 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993900+0000", 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013050099999999999 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.267 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998508+0000", 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046072220000000002 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:78 2.4 2:2f3e3e19:::rbd_data.13013c9ae692.0000000000000007:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.993772+0000", 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2675952559999999, 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0071295330000000004, 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.268 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 78 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993772+0000", 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993772+0000", 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993773+0000", 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1099999999999995e-07 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993797+0000", 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4490000000000001e-05 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993807+0000", 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3749999999999992e-06 2024-09-21T23:26:49.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993813+0000", 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4350000000000002e-06 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994193+0000", 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00038006500000000001 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994310+0000", 2024-09-21T23:26:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116495 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.000902+0000", 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065920620000000001 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:80 2.4 2:28f58da5:::rbd_data.13013c9ae692.0000000000000009:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.993829+0000", 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2675388500000002, 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0071714470000000001, 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 80 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993829+0000", 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.273 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993829+0000", 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993829+0000", 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6400000000000002e-07 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993834+0000", 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2570000000000001e-06 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993841+0000", 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2930000000000003e-06 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993846+0000", 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0860000000000001e-06 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994567+0000", 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00072146099999999996 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994673+0000", 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106118 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001000+0000", 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0063266679999999997 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:83 2.4 2:2574ce61:::rbd_data.13013c9ae692.000000000000000c:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.276 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.997889+0000", 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2634784669999997, 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0057942690000000003, 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 83 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997889+0000", 2024-09-21T23:26:49.277 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997889+0000", 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997890+0000", 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.246e-06 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.278 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997899+0000", 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1960000000000008e-06 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997909+0000", 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0869e-05 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997929+0000", 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9565999999999999e-05 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.279 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997982+0000", 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2680999999999999e-05 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998118+0000", 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013619399999999999 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003683+0000", 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055655169999999999 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.280 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:86 2.4 2:3fcce9c5:::rbd_data.13013c9ae692.000000000000000f:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.998646+0000", 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2627214530000002, 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051920330000000004, 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 86 2024-09-21T23:26:49.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998646+0000", 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998646+0000", 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.282 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998647+0000", 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7400000000000002e-07 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998652+0000", 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7879999999999997e-06 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998663+0000", 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1438e-05 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.283 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998671+0000", 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9950000000000005e-06 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998727+0000", 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5856000000000002e-05 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998858+0000", 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013130699999999999 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003838+0000", 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049797749999999997 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.2 epoch 57/18 rep_tid 65 entries 57'53 (0'0) error 2:4dd0ab61:::rbd_data.13013c9ae692.0000000000000013:head by client.5259.0:90 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'51)", 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.001575+0000", 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2597930450000003, 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046952690000000002, 2024-09-21T23:26:49.285 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001575+0000", 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001575+0000", 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001575+0000", 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1500000000000004e-07 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001583+0000", 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5739999999999998e-06 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001598+0000", 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5123000000000001e-05 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001607+0000", 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.3589999999999997e-06 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.001685+0000", 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8176000000000005e-05 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006270+0000", 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045844220000000003 2024-09-21T23:26:49.288 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:92 2.4 2:295f8ccd:::rbd_data.13013c9ae692.0000000000000015:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.003886+0000", 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2574810699999999, 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051605130000000003, 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.289 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 92 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003886+0000", 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003886+0000", 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.290 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003887+0000", 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.018e-06 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003892+0000", 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.831e-06 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003911+0000", 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8564e-05 2024-09-21T23:26:49.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003919+0000", 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6000000000000007e-06 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003982+0000", 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2848999999999994e-05 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004097+0000", 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000114855 2024-09-21T23:26:49.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009047+0000", 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049497960000000002 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:95 2.4 2:245a1e52:::rbd_data.13013c9ae692.0000000000000018:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.004138+0000", 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2572298210000001, 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049644850000000003, 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.293 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 95 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004138+0000", 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.294 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004138+0000", 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004139+0000", 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2600000000000001e-07 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004143+0000", 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4290000000000003e-06 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004153+0000", 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0261000000000001e-05 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004161+0000", 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2830000000000004e-06 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004346+0000", 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00018584299999999999 2024-09-21T23:26:49.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.004470+0000", 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012406900000000001 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009102+0000", 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004631774 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:100 2.4 2:3c2bac98:::rbd_data.13013c9ae692.000000000000001d:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.006738+0000", 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2546295049999996, 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053749649999999998, 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 100 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.298 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006738+0000", 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006738+0000", 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006739+0000", 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6300000000000004e-07 2024-09-21T23:26:49.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006742+0000", 2024-09-21T23:26:49.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3230000000000002e-06 2024-09-21T23:26:49.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.300 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.300 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006752+0000", 2024-09-21T23:26:49.300 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.329e-06 2024-09-21T23:26:49.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006756+0000", 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6609999999999999e-06 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006815+0000", 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8683e-05 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.301 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.006937+0000", 2024-09-21T23:26:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000122309 2024-09-21T23:26:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.012113+0000", 2024-09-21T23:26:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051757970000000002 2024-09-21T23:26:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.0 epoch 57/18 rep_tid 171 entries 57'3 (0'0) error 2:11991a95:::rbd_data.13013c9ae692.000000000000001f:head by client.5259.0:102 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'1)", 2024-09-21T23:26:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.009477+0000", 2024-09-21T23:26:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2518902819999997, 2024-09-21T23:26:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052640889999999996, 2024-09-21T23:26:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.303 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009477+0000", 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009477+0000", 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009478+0000", 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.23e-07 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.304 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009485+0000", 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1400000000000002e-06 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009501+0000", 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5699999999999999e-05 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009510+0000", 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0240000000000003e-06 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009568+0000", 2024-09-21T23:26:49.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8111e-05 2024-09-21T23:26:49.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.306 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.306 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.014741+0000", 2024-09-21T23:26:49.306 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051733910000000003 2024-09-21T23:26:49.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.306 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.2 epoch 57/18 rep_tid 68 entries 57'55 (0'0) error 2:58bc8c56:::rbd_data.13013c9ae692.000000000000001e:head by client.5259.0:101 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'53)", 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.009537+0000", 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2518301269999998, 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052335230000000003, 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009537+0000", 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009537+0000", 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009538+0000", 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.84e-07 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009541+0000", 2024-09-21T23:26:49.308 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.749e-06 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009555+0000", 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.366e-05 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009563+0000", 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2230000000000007e-06 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.009630+0000", 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6632999999999997e-05 2024-09-21T23:26:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.014771+0000", 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051409740000000004 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:99 2.4 2:22c0de70:::rbd_data.1286a02efebc.000000000000001c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.312522+0000", 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.948845114, 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046714399999999998, 2024-09-21T23:26:49.310 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 99 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312522+0000", 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.311 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312522+0000", 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312523+0000", 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7100000000000002e-07 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312526+0000", 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.9950000000000001e-06 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312533+0000", 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2880000000000004e-06 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312538+0000", 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.0479999999999998e-06 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313398+0000", 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00085969099999999999 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313512+0000", 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011357399999999999 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.317194+0000", 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036823730000000001 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:117 2.4 2:3053cb30:::journal_data.2.1286a02efebc.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+full_force+supports_pool_eio e57)", 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.326351+0000", 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.9350167489999999, 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049954220000000002, 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 117 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326351+0000", 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326351+0000", 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326352+0000", 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.009e-06 2024-09-21T23:26:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326356+0000", 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3839999999999999e-06 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326367+0000", 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0893999999999999e-05 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326375+0000", 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3189999999999995e-06 2024-09-21T23:26:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326428+0000", 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2286e-05 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.326475+0000", 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.7434e-05 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.331346+0000", 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048710960000000001 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5313.0:132 2.4 2:23a8bb95:::rbd_data.135878064453.000000000000003d:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.851577+0000", 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.4097906440000001, 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047780490000000004, 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5313", 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/2898988500", 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 132 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.851577+0000", 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.851577+0000", 2024-09-21T23:26:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.851577+0000", 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9600000000000001e-07 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.851579+0000", 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4249999999999999e-06 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.851583+0000", 2024-09-21T23:26:49.321 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0790000000000002e-06 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.851586+0000", 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.675e-06 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.854148+0000", 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0025618329999999999 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.322 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.854265+0000", 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116796 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.856355+0000", 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002090045 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5373.0:79 2.4 2:3436030f:::rbd_data.138b5612b65e.0000000000000008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.396246+0000", 2024-09-21T23:26:49.323 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.8651217870000001, 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0068705520000000003, 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5373", 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 79 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396246+0000", 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396246+0000", 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396247+0000", 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.263e-06 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396253+0000", 2024-09-21T23:26:49.325 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6370000000000004e-06 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396269+0000", 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6013000000000001e-05 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396276+0000", 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7849999999999992e-06 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396341+0000", 2024-09-21T23:26:49.326 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4624999999999999e-05 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396479+0000", 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013839599999999999 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403116+0000", 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.006636833 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.0 epoch 57/18 rep_tid 232 entries 57'28 (0'0) error 2:05e4dccf:::rbd_data.138b5612b65e.000000000000000a:head by client.5373.0:81 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'26)", 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.398033+0000", 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.8633346450000001, 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004850287, 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398033+0000", 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.328 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398033+0000", 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398034+0000", 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.64e-07 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398043+0000", 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5789999999999993e-06 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.329 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398066+0000", 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2541999999999999e-05 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398082+0000", 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5766000000000001e-05 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398164+0000", 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2238999999999996e-05 2024-09-21T23:26:49.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.402883+0000", 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047191969999999996 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5373.0:92 2.4 2:3e34f002:::rbd_data.138b5612b65e.0000000000000014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.402990+0000", 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.8583778739999999, 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0053275989999999997, 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.331 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5373", 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 92 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.402990+0000", 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.332 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.402990+0000", 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.402991+0000", 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2079999999999999e-06 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.402995+0000", 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3270000000000002e-06 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.333 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403005+0000", 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8209999999999997e-06 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403012+0000", 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6789999999999999e-06 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403185+0000", 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00017350199999999999 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.334 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403323+0000", 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013756900000000001 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.408317+0000", 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004994493 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5373.0:96 2.4 2:3bc17cd4:::rbd_data.138b5612b65e.0000000000000018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.335 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.403284+0000", 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.8580839620000003, 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005112274, 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5373", 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 96 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.336 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403284+0000", 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403284+0000", 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403284+0000", 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.71e-07 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.337 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403289+0000", 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4399999999999998e-06 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403300+0000", 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1318000000000001e-05 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403307+0000", 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9809999999999997e-06 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403631+0000", 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00032393299999999999 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403751+0000", 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011984100000000001 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.408396+0000", 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046448899999999996 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.339 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5373.0:110 2.4 2:20eb86a2:::rbd_data.138b5612b65e.0000000000000026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.409551+0000", 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.851816677, 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0062934410000000003, 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5373", 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 110 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.340 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.409551+0000", 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.409551+0000", 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.409552+0000", 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4600000000000003e-07 2024-09-21T23:26:49.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.409556+0000", 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5539999999999999e-06 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.409568+0000", 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1325000000000001e-05 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.409575+0000", 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9829999999999999e-06 2024-09-21T23:26:49.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.409623+0000", 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8306999999999997e-05 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.409738+0000", 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000114948 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.343 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415844+0000", 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0061064780000000003 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.2 epoch 57/18 rep_tid 155 entries 57'100 (0'0) error 2:4cf829bc:::rbd_data.138b5612b65e.0000000000000035:head by client.5373.0:125 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'95)", 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.416252+0000", 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 2.8451156659999999, 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046721890000000002, 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.416252+0000", 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.416252+0000", 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.345 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.416253+0000", 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3099999999999997e-07 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.416269+0000", 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6175999999999999e-05 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.416287+0000", 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8521000000000001e-05 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.346 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.416304+0000", 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6529999999999999e-05 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.416378+0000", 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3794000000000005e-05 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.420924+0000", 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045464370000000004 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.348 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.348 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:49.348 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6: {} 2024-09-21T23:26:49.349 INFO:teuthology.orchestra.run.smithi031.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T23:26:49.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T23:26:49.350 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7: {} 2024-09-21T23:26:49.351 INFO:teuthology.orchestra.run.smithi031.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T23:26:49.353 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete...2024-09-21T23:26:49.344+0000 7f45815e0640 0 -- 172.21.15.31:0/4002515302 >> [v2:172.21.15.60:6800/3490482162,v1:172.21.15.60:6801/3490482162] conn(0x7f456000f6d0 msgr2=0x7f4560011ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-21T23:26:49.353 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-21T23:26:49.348+0000 7f16c3d7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33569) UID: 0 2024-09-21T23:26:49.359 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete...2024-09-21T23:26:49.359+0000 7f4582842640 0 -- 172.21.15.31:0/4002515302 >> [v2:172.21.15.60:6816/1315554966,v1:172.21.15.60:6817/1315554966] conn(0x7f456000bf30 msgr2=0x7f456000e320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-21T23:26:49.449 INFO:tasks.ceph.osd.5.smithi060.stderr:2024-09-21T23:26:49.449+0000 7f5814770640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33463) UID: 0 2024-09-21T23:26:49.464 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:49.465 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:49.465 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:49.473 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T23:26:49.550 INFO:tasks.ceph.osd.6.smithi060.stderr:2024-09-21T23:26:49.549+0000 7fb011bdf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33469) UID: 0 2024-09-21T23:26:49.574 INFO:tasks.workunit.client.0.smithi031.stderr: Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete... Removing image: 100% complete...done. 2024-09-21T23:26:49.578 INFO:tasks.workunit.client.0.smithi031.stderr:+ set +x 2024-09-21T23:26:49.578 INFO:tasks.workunit.client.0.smithi031.stdout:OK 2024-09-21T23:26:49.610 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T23:26:49.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T23:26:49.625 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:49.626 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:49.626 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:49.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T23:26:49.638 INFO:tasks.workunit:Stopping ['rbd/journal.sh'] on client.0... 2024-09-21T23:26:49.638 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T23:26:49.681 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:49.680+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:49.752 INFO:tasks.ceph.osd.7.smithi060.stderr:2024-09-21T23:26:49.752+0000 7f9fbf7b6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33467) UID: 0 2024-09-21T23:26:49.789 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5244.0:61 2.1 2:83b64d0f:::rbd_header.13013c9ae692:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.891124+0000", 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.8965477640000001, 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00466922, 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5244", 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3352402684", 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 61 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891124+0000", 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891124+0000", 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891126+0000", 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1100000000000001e-06 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891137+0000", 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0474e-05 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891161+0000", 2024-09-21T23:26:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.4879e-05 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891173+0000", 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1708e-05 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891254+0000", 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0748999999999995e-05 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op must wait for map", 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.891261+0000", 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7780000000000002e-06 2024-09-21T23:26:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.893987+0000", 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002726235 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.894068+0000", 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1336999999999997e-05 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3", 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.894394+0000", 2024-09-21T23:26:49.794 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000325827 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.895449+0000", 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0010548510000000001 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.895674+0000", 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000225474 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:49.795 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.895739+0000", 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.4941e-05 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.895793+0000", 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3857000000000003e-05 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:77 2.1 2:9b2c1414:::rbd_data.13013c9ae692.0000000000000006:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.993724+0000", 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.7939473249999995, 2024-09-21T23:26:49.796 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049854790000000001, 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 77 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993724+0000", 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.797 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993724+0000", 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993725+0000", 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4700000000000002e-07 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993728+0000", 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.0120000000000001e-06 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.798 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993736+0000", 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9249999999999995e-06 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.993741+0000", 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7039999999999999e-06 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994202+0000", 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00046076199999999998 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994327+0000", 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012460800000000001 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998710+0000", 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004383021 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:84 2.1 2:987fd1fb:::rbd_data.13013c9ae692.000000000000000d:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.800 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.997932+0000", 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.7897399309999997, 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050909830000000003, 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 84 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997932+0000", 2024-09-21T23:26:49.801 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997932+0000", 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997933+0000", 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.235e-06 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.802 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997938+0000", 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4020000000000001e-06 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997953+0000", 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4986e-05 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997962+0000", 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3310000000000004e-06 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.803 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998021+0000", 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.889e-05 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998141+0000", 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000120043 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003023+0000", 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048820959999999998 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.804 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.3 epoch 57/18 rep_tid 64 entries 57'15 (0'0) error 2:cfc8b22f:::rbd_data.13013c9ae692.000000000000000e:head by client.5259.0:85 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'13)", 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.998638+0000", 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.7890332319999995, 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050258189999999999, 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.805 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998638+0000", 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998638+0000", 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998639+0000", 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9999999999999997e-07 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.806 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998647+0000", 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.9100000000000005e-06 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998660+0000", 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.314e-05 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998669+0000", 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3240000000000003e-06 2024-09-21T23:26:49.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998756+0000", 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7517999999999998e-05 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003664+0000", 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049082270000000002 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.808 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:87 2.1 2:834d47d4:::rbd_data.13013c9ae692.0000000000000010:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.998828+0000", 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.788843484, 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.005133398, 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 87 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.809 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998828+0000", 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998828+0000", 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998829+0000", 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4399999999999998e-07 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998838+0000", 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.5590000000000006e-06 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998852+0000", 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4072e-05 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998860+0000", 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7349999999999996e-06 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.811 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998916+0000", 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6033999999999999e-05 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998990+0000", 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4908999999999997e-05 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003962+0000", 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049711449999999997 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.812 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5259.0:121 2.1 2:8afe7a84:::journal.13013c9ae692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.212175+0000", 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.5754971580000001, 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048095250000000003, 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5259", 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:49.813 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 121 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212175+0000", 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212175+0000", 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.814 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212175+0000", 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8500000000000002e-07 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212178+0000", 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7350000000000001e-06 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212184+0000", 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.6170000000000003e-06 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.815 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212188+0000", 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.6229999999999999e-06 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212481+0000", 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000292472 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212567+0000", 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6602999999999997e-05 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "waiting for subops from 3", 2024-09-21T23:26:49.816 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212679+0000", 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000111721 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216910+0000", 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0042315629999999998 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "op_commit", 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216934+0000", 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3595999999999999e-05 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.817 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216959+0000", 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5014000000000001e-05 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216984+0000", 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.5096000000000002e-05 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:98 2.1 2:97617b74:::rbd_data.1286a02efebc.000000000000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.818 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.312348+0000", 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.4753236779999996, 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004809598, 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 98 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.819 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312348+0000", 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312348+0000", 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312349+0000", 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.78e-07 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.820 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312356+0000", 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.691e-06 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312366+0000", 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0708e-05 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312375+0000", 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.7199999999999995e-06 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.821 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312539+0000", 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00016393700000000001 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312646+0000", 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000106838 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.317158+0000", 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045117259999999998 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.822 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.4946.0:100 2.1 2:8eef717c:::rbd_data.1286a02efebc.000000000000001d:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.312634+0000", 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.4750372609999998, 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046341109999999998, 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.4946", 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 100 2024-09-21T23:26:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312634+0000", 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312634+0000", 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312635+0000", 2024-09-21T23:26:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.1699999999999997e-07 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312648+0000", 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.225e-05 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312657+0000", 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.0929999999999992e-06 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312661+0000", 2024-09-21T23:26:49.825 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6750000000000001e-06 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313023+0000", 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00036178000000000002 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313132+0000", 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000109287 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.317269+0000", 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0041361089999999998 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5373.0:80 2.1 2:8d7a4cd3:::rbd_data.138b5612b65e.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.396328+0000", 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.3913436620000001, 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0065562270000000004, 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.827 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5373", 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 80 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396328+0000", 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396328+0000", 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396329+0000", 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5899999999999996e-07 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396332+0000", 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4149999999999999e-06 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396340+0000", 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.994e-06 2024-09-21T23:26:49.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396346+0000", 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.7769999999999999e-06 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396786+0000", 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00044013899999999998 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396897+0000", 2024-09-21T23:26:49.830 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011069300000000001 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.402884+0000", 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0059876499999999997 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5373.0:83 2.1 2:9679c393:::rbd_data.138b5612b65e.000000000000000b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.397826+0000", 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.389845555, 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0051745669999999997, 2024-09-21T23:26:49.831 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5373", 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 83 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397826+0000", 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397826+0000", 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397827+0000", 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2699999999999998e-07 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397832+0000", 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9389999999999996e-06 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397846+0000", 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4303e-05 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397854+0000", 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.537e-06 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397908+0000", 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3838999999999997e-05 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398047+0000", 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00013940600000000001 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403001+0000", 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049536160000000001 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5373.0:95 2.6 2:7ece13d1:::rbd_data.138b5612b65e.0000000000000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.403257+0000", 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.3844146610000001, 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0052899490000000004, 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5373", 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 95 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403257+0000", 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403257+0000", 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403258+0000", 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2500000000000004e-07 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.837 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403262+0000", 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4959999999999998e-06 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403276+0000", 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3399e-05 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403283+0000", 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2779999999999996e-06 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.838 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.839 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403343+0000", 2024-09-21T23:26:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9843000000000003e-05 2024-09-21T23:26:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403475+0000", 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000132058 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.408547+0000", 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0050720499999999998 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.3 epoch 57/18 rep_tid 145 entries 57'55 (0'0) error 2:c9bf251e:::rbd_data.138b5612b65e.0000000000000019:head by client.5373.0:97 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'52)", 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.403770+0000", 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.383902086, 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004664975, 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403770+0000", 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.913 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403770+0000", 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403770+0000", 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.9699999999999996e-07 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403785+0000", 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4799e-05 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403807+0000", 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.1675000000000002e-05 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403823+0000", 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5971e-05 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.403933+0000", 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000110482 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.408435+0000", 2024-09-21T23:26:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045014510000000001 2024-09-21T23:26:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5373.0:113 2.1 2:8d3177d6:::rbd_data.138b5612b65e.0000000000000029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.410930+0000", 2024-09-21T23:26:49.916 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.3767421280000001, 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049423480000000001, 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5373", 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 113 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.917 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.410930+0000", 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.410930+0000", 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.410930+0000", 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.9100000000000004e-07 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.410933+0000", 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7020000000000001e-06 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.410938+0000", 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.5119999999999999e-06 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.410942+0000", 2024-09-21T23:26:49.919 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.4649999999999999e-06 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411241+0000", 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00029923999999999999 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411359+0000", 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011755 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415872+0000", 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0045133880000000001 2024-09-21T23:26:49.920 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5373.0:120 2.1 2:94fb10ec:::rbd_data.138b5612b65e.0000000000000030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.413811+0000", 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.373860911, 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046008810000000002, 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5373", 2024-09-21T23:26:49.921 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 120 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.413811+0000", 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.413811+0000", 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.413812+0000", 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.71e-07 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.413817+0000", 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.3240000000000002e-06 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.413829+0000", 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1535999999999999e-05 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.923 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.413836+0000", 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5050000000000001e-06 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414225+0000", 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00038906200000000002 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414341+0000", 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000116084 2024-09-21T23:26:49.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.418412+0000", 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004070499 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5580.0:82 2.1 2:9f171020:::rbd_data.15994fe581e3.000000000000000b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:49.346445+0000", 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.44122668399999998, 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049249610000000003, 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.925 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5580", 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 82 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346445+0000", 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.926 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346445+0000", 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346446+0000", 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0759999999999999e-06 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346451+0000", 2024-09-21T23:26:49.927 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.5449999999999997e-06 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346465+0000", 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4245e-05 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346473+0000", 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4230000000000005e-06 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346543+0000", 2024-09-21T23:26:49.928 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9339999999999997e-05 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346631+0000", 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.8046000000000004e-05 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.351370+0000", 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047392859999999997 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5580.0:83 2.6 2:7cb60cd4:::rbd_data.15994fe581e3.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:49.346550+0000", 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.44112190099999998, 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048191179999999998, 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5580", 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 83 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.930 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346550+0000", 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346550+0000", 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346551+0000", 2024-09-21T23:26:49.931 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.9600000000000008e-07 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346558+0000", 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9569999999999996e-06 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346574+0000", 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6467999999999999e-05 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346586+0000", 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1749999999999999e-05 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346629+0000", 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.3300000000000002e-05 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346697+0000", 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7979999999999996e-05 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.933 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.351369+0000", 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004671667 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5580.0:84 2.6 2:64285eac:::rbd_data.15994fe581e3.000000000000000d:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:49.346604+0000", 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.44106757099999999, 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048447009999999999, 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.934 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5580", 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 84 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346604+0000", 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.935 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346604+0000", 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346605+0000", 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.1500000000000004e-07 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346608+0000", 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.3749999999999999e-06 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.936 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346615+0000", 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2740000000000004e-06 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346621+0000", 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3550000000000001e-06 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346989+0000", 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00036848000000000002 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.937 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.347117+0000", 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00012798400000000001 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.351449+0000", 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043315180000000003 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5580.0:85 2.6 2:6278c85b:::rbd_data.15994fe581e3.000000000000000e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:49.938 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:49.346635+0000", 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.44103711600000001, 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0048924930000000004, 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5580", 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 85 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.939 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346635+0000", 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346635+0000", 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346635+0000", 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4700000000000002e-07 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.940 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346638+0000", 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7410000000000001e-06 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346645+0000", 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7279999999999998e-06 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.346651+0000", 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3389999999999998e-06 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.941 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.347390+0000", 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00073924199999999998 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.347475+0000", 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4950000000000005e-05 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.351527+0000", 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040520460000000001 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.942 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5580.0:101 2.1 2:91f15b7f:::rbd_data.15994fe581e3.000000000000001e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:49.354820+0000", 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.43285129, 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004820693, 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5580", 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:49.943 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 101 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354820+0000", 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354820+0000", 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.944 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354821+0000", 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6799999999999999e-07 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354826+0000", 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.4889999999999997e-06 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354839+0000", 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.344e-05 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.945 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354847+0000", 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7239999999999992e-06 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354902+0000", 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4746000000000001e-05 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.355030+0000", 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000127911 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.946 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.359641+0000", 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0046115150000000001 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_op(client.5580.0:125 2.6 2:7e772d97:::rbd_data.15994fe581e3.0000000000000036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:49.378095+0000", 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 0.40957685100000002, 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0049495110000000002, 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:49.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "started", 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "client_info": { 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "client": "client.5580", 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "tid": 125 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.378095+0000", 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.378095+0000", 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:49.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.378096+0000", 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.2600000000000002e-07 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.378100+0000", 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0330000000000002e-06 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.378109+0000", 2024-09-21T23:26:49.949 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.5089999999999999e-06 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.378126+0000", 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6410999999999999e-05 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.378172+0000", 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.6217000000000002e-05 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.378305+0000", 2024-09-21T23:26:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.000133633 2024-09-21T23:26:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.383044+0000", 2024-09-21T23:26:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0047389820000000001 2024-09-21T23:26:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:49.951 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:49.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T23:26:49.953 INFO:tasks.ceph.osd.4.smithi060.stderr:2024-09-21T23:26:49.951+0000 7fb33a356640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33461) UID: 0 2024-09-21T23:26:49.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T23:26:49.962 DEBUG:teuthology.parallel:result is None 2024-09-21T23:26:49.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T23:26:50.029 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T23:26:50.029 DEBUG:teuthology.orchestra.run.smithi031:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T23:26:50.069 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-21T23:26:50.069+0000 7f60e0da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33571) UID: 0 2024-09-21T23:26:50.094 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T23:26:50.094 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-21T23:26:50.103 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T23:26:50.114 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: "num_ops": 0 2024-09-21T23:26:50.114 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:50.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T23:26:50.272 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [], 2024-09-21T23:26:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "complaint_time": 30, 2024-09-21T23:26:50.272 INFO:teuthology.orchestra.run.smithi031.stdout: "num_blocked_ops": 0 2024-09-21T23:26:50.272 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:50.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T23:26:50.389 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:26:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "size": 20, 2024-09-21T23:26:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 600, 2024-09-21T23:26:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "ops": [ 2024-09-21T23:26:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.389 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 55 entries 57'54 (0'0) error 2:9b2c1414:::rbd_data.13013c9ae692.0000000000000006:head by client.5259.0:77 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'51)", 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.994604+0000", 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 5.3936383719999998, 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033897520000000002, 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994604+0000", 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994604+0000", 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994605+0000", 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9999999999999997e-07 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994613+0000", 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0800000000000006e-06 2024-09-21T23:26:50.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994631+0000", 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7530999999999999e-05 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994645+0000", 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.4372999999999999e-05 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.994742+0000", 2024-09-21T23:26:50.392 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.7409000000000002e-05 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.997994+0000", 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0032516590000000001 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 56 entries 57'55 (0'0) error 2:987fd1fb:::rbd_data.13013c9ae692.000000000000000d:head by client.5259.0:84 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'52)", 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.998421+0000", 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 5.3898214959999997, 2024-09-21T23:26:50.393 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00229803, 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998421+0000", 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998421+0000", 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998422+0000", 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.4900000000000004e-07 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998431+0000", 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4849999999999992e-06 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998453+0000", 2024-09-21T23:26:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2082999999999998e-05 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998472+0000", 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9490000000000001e-05 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.998552+0000", 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0091000000000003e-05 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.000719+0000", 2024-09-21T23:26:50.396 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021669319999999999 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 57 entries 57'56 (0'0) error 2:834d47d4:::rbd_data.13013c9ae692.0000000000000010:head by client.5259.0:87 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'53)", 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:44.999182+0000", 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 5.3890609529999995, 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044732239999999996, 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999182+0000", 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999182+0000", 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999182+0000", 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8400000000000004e-07 2024-09-21T23:26:50.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999191+0000", 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3529999999999995e-06 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999213+0000", 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2252e-05 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.399 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999229+0000", 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5973000000000001e-05 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:44.999303+0000", 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3640000000000006e-05 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.003655+0000", 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0043523219999999996 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.5259.0:116 2.1 e57/18 2:9e43db55:::journal_data.2.13013c9ae692.0:head v 57'63, mlcod=57'61)", 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.209475+0000", 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 5.1787675740000001, 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0019873709999999999, 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209475+0000", 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209475+0000", 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209476+0000", 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5799999999999998e-07 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209482+0000", 2024-09-21T23:26:50.402 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6890000000000001e-06 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209493+0000", 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1806e-05 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209500+0000", 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.8700000000000003e-06 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.403 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209553+0000", 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2355000000000003e-05 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.209573+0000", 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0219000000000001e-05 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.211393+0000", 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.001819745 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.404 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.211462+0000", 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9929000000000001e-05 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.5259.0:121 2.1 e57/18 2:8afe7a84:::journal.13013c9ae692:head v 57'65, mlcod=57'62)", 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.212741+0000", 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 5.1755015310000001, 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040698150000000001, 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.405 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212741+0000", 2024-09-21T23:26:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212741+0000", 2024-09-21T23:26:50.406 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212742+0000", 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4900000000000005e-07 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212758+0000", 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6032000000000002e-05 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.407 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212769+0000", 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.095e-05 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212775+0000", 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6500000000000001e-06 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212827+0000", 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2098000000000002e-05 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.212838+0000", 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1165000000000001e-05 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216768+0000", 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039302679999999998 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.216811+0000", 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.2902999999999999e-05 2024-09-21T23:26:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 76 entries 57'69 (0'0) error 2:873b27ee:::rbd_data.1286a02efebc.0000000000000013:head by client.4946.0:90 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'66)", 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.310738+0000", 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 5.077504835, 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021035989999999998, 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310738+0000", 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310738+0000", 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310738+0000", 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4499999999999997e-07 2024-09-21T23:26:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310745+0000", 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.9160000000000004e-06 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310762+0000", 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6240999999999999e-05 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310790+0000", 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.8316000000000001e-05 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.310848+0000", 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8136e-05 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312841+0000", 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0019934449999999999 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 78 entries 57'71 (0'0) error 2:97617b74:::rbd_data.1286a02efebc.000000000000001b:head by client.4946.0:98 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'67)", 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.312857+0000", 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 5.0753852569999998, 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040794789999999996, 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312857+0000", 2024-09-21T23:26:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312857+0000", 2024-09-21T23:26:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312858+0000", 2024-09-21T23:26:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1200000000000003e-07 2024-09-21T23:26:50.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312866+0000", 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.3000000000000002e-06 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312884+0000", 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7923000000000001e-05 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.416 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312898+0000", 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3322999999999999e-05 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.312980+0000", 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2876999999999999e-05 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.316937+0000", 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039564439999999999 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.417 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 79 entries 57'72 (0'0) error 2:8eef717c:::rbd_data.1286a02efebc.000000000000001d:head by client.4946.0:100 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'67)", 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:45.313337+0000", 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 5.0749060459999997, 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0036166739999999998, 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313337+0000", 2024-09-21T23:26:50.418 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313337+0000", 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313337+0000", 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0999999999999998e-07 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.419 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313345+0000", 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8599999999999993e-06 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313366+0000", 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0801e-05 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313382+0000", 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5877000000000001e-05 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.420 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.313482+0000", 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010050799999999999 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:45.316953+0000", 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0034710180000000002 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "osd_repop(client.5328.0:52 2.1 e57/18 2:9c628179:::rbd_id.testrbdcopys46970:head v 57'78, mlcod=57'76)", 2024-09-21T23:26:50.421 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.113617+0000", 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 4.2746258749999999, 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020936610000000001, 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113617+0000", 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.422 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113617+0000", 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113618+0000", 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2309999999999999e-06 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113626+0000", 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2819999999999996e-06 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.423 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113643+0000", 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6164e-05 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113650+0000", 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.3939999999999998e-06 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113713+0000", 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.3495000000000001e-05 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "started", 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.113737+0000", 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3142999999999998e-05 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "commit_sent", 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.115662+0000", 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0019258840000000001 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.115710+0000", 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.8068000000000001e-05 2024-09-21T23:26:50.425 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 96 entries 57'79 (0'0) error 2:9f98d0d9:::rbd_data.138b5612b65e.0000000000000007:head by client.5373.0:78 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'77)", 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.396762+0000", 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.991480932, 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033279669999999998, 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.426 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396762+0000", 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396762+0000", 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396763+0000", 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6449999999999999e-06 2024-09-21T23:26:50.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396776+0000", 2024-09-21T23:26:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.279e-05 2024-09-21T23:26:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396814+0000", 2024-09-21T23:26:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.7703999999999997e-05 2024-09-21T23:26:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396832+0000", 2024-09-21T23:26:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.8117e-05 2024-09-21T23:26:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.396946+0000", 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00011387000000000001 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.400090+0000", 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0031438410000000001 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 98 entries 57'80 (0'0) error 2:8d7a4cd3:::rbd_data.138b5612b65e.0000000000000009:head by client.5373.0:80 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'77)", 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.397140+0000", 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.991103072, 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0055420900000000004, 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397140+0000", 2024-09-21T23:26:50.431 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397140+0000", 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397140+0000", 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.2799999999999996e-07 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397148+0000", 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2730000000000002e-06 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397166+0000", 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7569999999999999e-05 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397181+0000", 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.453e-05 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.433 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.397270+0000", 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.9159999999999993e-05 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.402682+0000", 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0054120289999999996 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 101 entries 57'81 (0'0) error 2:9679c393:::rbd_data.138b5612b65e.000000000000000b:head by client.5373.0:83 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'77)", 2024-09-21T23:26:50.434 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.398263+0000", 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.9899801049999999, 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0044348599999999997, 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398263+0000", 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.435 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398263+0000", 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398263+0000", 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.4600000000000003e-07 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398275+0000", 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.1294000000000001e-05 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.436 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398297+0000", 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.2517000000000001e-05 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398312+0000", 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5223e-05 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.398394+0000", 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.1848999999999995e-05 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.402697+0000", 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.004303131 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 105 entries 57'82 (0'0) error 2:979d07b6:::rbd_data.138b5612b65e.000000000000001c:head by client.5373.0:100 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'80)", 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.406175+0000", 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.9820674660000002, 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002193515, 2024-09-21T23:26:50.438 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406175+0000", 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406175+0000", 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.439 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406179+0000", 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 4.0419999999999996e-06 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406190+0000", 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.0848e-05 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406217+0000", 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.7067e-05 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406234+0000", 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7011999999999999e-05 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.406310+0000", 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.5791000000000007e-05 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.408369+0000", 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020587549999999998 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.441 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 107 entries 57'84 (0'0) error 2:8d3177d6:::rbd_data.138b5612b65e.0000000000000029:head by client.5373.0:113 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'81)", 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.411565+0000", 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.9766780179999999, 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040547370000000001, 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.442 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411565+0000", 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411565+0000", 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411565+0000", 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2099999999999996e-07 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.443 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411573+0000", 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.0020000000000006e-06 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411590+0000", 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6986999999999998e-05 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411603+0000", 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.3169999999999999e-05 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.411705+0000", 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00010180500000000001 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415619+0000", 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0039141519999999997 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.445 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 109 entries 57'85 (0'0) error 2:99397b72:::rbd_data.138b5612b65e.000000000000002f:head by client.5373.0:119 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'82)", 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.414176+0000", 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.9740662850000001, 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0038963309999999998, 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414176+0000", 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.446 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414176+0000", 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414177+0000", 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.8800000000000002e-07 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414186+0000", 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.2129999999999997e-06 2024-09-21T23:26:50.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414210+0000", 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.3988000000000001e-05 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414227+0000", 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.6714999999999998e-05 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414325+0000", 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.8095000000000002e-05 2024-09-21T23:26:50.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.418073+0000", 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003747532 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 110 entries 57'86 (0'0) error 2:94fb10ec:::rbd_data.138b5612b65e.0000000000000030:head by client.5373.0:120 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'82)", 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.414564+0000", 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.973678853, 2024-09-21T23:26:50.449 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0035261559999999999, 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414564+0000", 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414564+0000", 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414564+0000", 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2099999999999996e-07 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414572+0000", 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.7060000000000004e-06 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414593+0000", 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 2.0902000000000001e-05 2024-09-21T23:26:50.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414608+0000", 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.446e-05 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.414694+0000", 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.6805999999999995e-05 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.418090+0000", 2024-09-21T23:26:50.452 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0033956609999999999 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 57/18 rep_tid 111 entries 57'87 (0'0) error 2:8b7e6962:::rbd_data.138b5612b65e.0000000000000032:head by client.5373.0:122 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'82)", 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:46.415021+0000", 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 3.9732216579999999, 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.003083026, 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415021+0000", 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415021+0000", 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415022+0000", 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.0100000000000005e-07 2024-09-21T23:26:50.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415029+0000", 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 7.4599999999999997e-06 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415047+0000", 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.7419e-05 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415059+0000", 2024-09-21T23:26:50.455 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.2211e-05 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.415113+0000", 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.4506999999999998e-05 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:46.418104+0000", 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0029908280000000001 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 59/18 rep_tid 197 entries 59'119 (0'0) error 2:9f14d298:::rbd_data.15994fe581e3.0000000000000010:head by client.5580.0:87 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 59'116)", 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:49.348421+0000", 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.0398214320000001, 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0028674009999999999, 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.348421+0000", 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.348421+0000", 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.348422+0000", 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.6700000000000003e-07 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.348429+0000", 2024-09-21T23:26:50.458 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.7399999999999998e-06 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.348444+0000", 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5454000000000001e-05 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.348454+0000", 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 9.6639999999999993e-06 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.459 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.348505+0000", 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.1536999999999998e-05 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.351289+0000", 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.002783439 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 59/18 rep_tid 200 entries 59'121 (0'0) error 2:8acbf0eb:::rbd_data.15994fe581e3.000000000000001a:head by client.5580.0:97 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 59'118)", 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:49.354579+0000", 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.0336634330000001, 2024-09-21T23:26:50.460 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0021315219999999998, 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354579+0000", 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354579+0000", 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354580+0000", 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.1999999999999999e-07 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354586+0000", 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.409e-06 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354602+0000", 2024-09-21T23:26:50.462 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.5852e-05 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354616+0000", 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.34e-05 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.354698+0000", 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.2770999999999997e-05 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.463 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.356711+0000", 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0020124700000000001 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "description": "pg_update_log_missing(2.1 epoch 59/18 rep_tid 201 entries 59'122 (0'0) error 2:91f15b7f:::rbd_data.15994fe581e3.000000000000001e:head by client.5580.0:101 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 59'119)", 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "initiated_at": "2024-09-21T23:26:49.355263+0000", 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "age": 1.032979581, 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.00415498, 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "continuous": false, 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type_data": { 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.464 INFO:teuthology.orchestra.run.smithi031.stdout: "events": [ 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "initiated", 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.355263+0000", 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "header_read", 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.355263+0000", 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "throttled", 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.355264+0000", 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 5.8100000000000003e-07 2024-09-21T23:26:50.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "all_read", 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.355272+0000", 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 8.089e-06 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "dispatched", 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.355291+0000", 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 1.9137e-05 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.355324+0000", 2024-09-21T23:26:50.466 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 3.2907999999999997e-05 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "reached_pg", 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.355386+0000", 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 6.2429999999999997e-05 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "event": "done", 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "time": "2024-09-21T23:26:49.359418+0000", 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: "duration": 0.0040318350000000001 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-21T23:26:50.467 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-09-21T23:26:50.468 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:26:50.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T23:26:50.589 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:50.589 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:50.589 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:50.589 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:50.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T23:26:50.706 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:50.706 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:50.706 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:50.706 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:50.706 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:50.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T23:26:50.823 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 51.520497828000003, 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00108676, 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301958+0000", 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.301959+0000", 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3099999999999996e-07 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302322+0000", 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00036268299999999998 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302345+0000", 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3278e-05 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302367+0000", 2024-09-21T23:26:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2118999999999999e-05 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302468+0000", 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100579 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.302479+0000", 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1430999999999999e-05 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.303045+0000", 2024-09-21T23:26:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000565739 2024-09-21T23:26:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPush(1.0 35/34 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T23:26:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 51.51774159, 2024-09-21T23:26:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010194869999999999, 2024-09-21T23:26:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304714+0000", 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.304715+0000", 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9599999999999998e-07 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305026+0000", 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031072100000000002 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305048+0000", 2024-09-21T23:26:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1875999999999999e-05 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305066+0000", 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8644e-05 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305163+0000", 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6333000000000001e-05 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305172+0000", 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2189999999999993e-06 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:59.305734+0000", 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056179799999999996 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4742.0:6 2.6 e54/42 2:61b297bc:::rbd_id.testrbdjournal46970:head v 54'1, mlcod=0'0)", 2024-09-21T23:26:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:41.854628+0000", 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 8.967827861, 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0009966459999999999, 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:41.854628+0000", 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:41.854628+0000", 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:41.854628+0000", 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.17e-07 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:41.854634+0000", 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.054e-06 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:41.854648+0000", 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3456e-05 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:41.854658+0000", 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9539999999999999e-06 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:41.854735+0000", 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7229000000000002e-05 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:41.854753+0000", 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8491999999999998e-05 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:41.855578+0000", 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00082496999999999996 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:41.855625+0000", 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6273999999999998e-05 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 53 entries 57'2 (0'0) error 2:62f7b169:::rbd_data.13013c9ae692.0000000000000003:head by client.5259.0:74 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:44.994181+0000", 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 5.828275111, 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00064976400000000005, 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.994181+0000", 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.994181+0000", 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.994181+0000", 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5999999999999999e-07 2024-09-21T23:26:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.994193+0000", 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1535999999999999e-05 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.994209+0000", 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6643999999999999e-05 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.994220+0000", 2024-09-21T23:26:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0885e-05 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.994308+0000", 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7497999999999995e-05 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.994830+0000", 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052274099999999996 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4946.0:46 2.6 e57/42 2:61b297bc:::rbd_id.testrbdjournal46970:head v 57'6, mlcod=57'4)", 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.289862+0000", 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 5.5325941250000001, 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000803454, 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.289862+0000", 2024-09-21T23:26:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.289862+0000", 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.289863+0000", 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6530000000000001e-06 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.289883+0000", 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9737000000000001e-05 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.289909+0000", 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6319999999999999e-05 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.289923+0000", 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3326000000000001e-05 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.290035+0000", 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011268300000000001 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.290075+0000", 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9375999999999997e-05 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.290634+0000", 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000559287 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.290665+0000", 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1072000000000003e-05 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 71 entries 57'7 (0'0) error 2:7a4ce2c7:::rbd_data.1286a02efebc.0000000000000007:head by client.4946.0:78 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'5)", 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.305543+0000", 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 5.5169123190000002, 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000627237, 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.305543+0000", 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.305543+0000", 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.305544+0000", 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1500000000000004e-07 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.305553+0000", 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7050000000000005e-06 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.305582+0000", 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9422000000000001e-05 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.305601+0000", 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8451e-05 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.305693+0000", 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1860999999999993e-05 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.306171+0000", 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00047808300000000002 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 81 entries 57'10 (0'0) error 2:69f25086:::rbd_data.135878064453.000000000000000b:head by client.5313.0:82 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'8)", 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.651055+0000", 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 5.171400642, 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00067910699999999997, 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.651055+0000", 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.651055+0000", 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.651056+0000", 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.221e-06 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.651069+0000", 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2337999999999999e-05 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.651097+0000", 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8212999999999999e-05 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.651126+0000", 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9181000000000001e-05 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.651237+0000", 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110816 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.651734+0000", 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049733799999999995 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 83 entries 57'11 (0'0) error 2:7450889d:::rbd_data.135878064453.000000000000000f:head by client.5313.0:86 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'9)", 2024-09-21T23:26:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.652361+0000", 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 5.1700943629999996, 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00074728700000000002, 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652361+0000", 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652361+0000", 2024-09-21T23:26:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652362+0000", 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6199999999999998e-07 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652368+0000", 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3409999999999999e-06 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652385+0000", 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6370000000000001e-05 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652393+0000", 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7099999999999996e-06 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652476+0000", 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2880000000000006e-05 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.653109+0000", 2024-09-21T23:26:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00063242400000000003 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 85 entries 57'12 (0'0) error 2:6d6f291b:::rbd_data.135878064453.0000000000000010:head by client.5313.0:87 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'9)", 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.652813+0000", 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 5.1696423879999998, 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00059531500000000002, 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652813+0000", 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652813+0000", 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652814+0000", 2024-09-21T23:26:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652822+0000", 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9910000000000002e-06 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652849+0000", 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6712000000000001e-05 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652864+0000", 2024-09-21T23:26:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5491000000000001e-05 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.652963+0000", 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8561999999999996e-05 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.653409+0000", 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00044598400000000002 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 91 entries 57'16 (0'0) error 2:67cae2ed:::rbd_mirroring:head by client.5313.0:157 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'14)", 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.878088+0000", 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.9443676649999997, 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00078250900000000003, 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.878088+0000", 2024-09-21T23:26:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.878088+0000", 2024-09-21T23:26:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.878090+0000", 2024-09-21T23:26:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.587e-06 2024-09-21T23:26:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.878107+0000", 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7139999999999999e-05 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.878150+0000", 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2706000000000001e-05 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.878174+0000", 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4349999999999999e-05 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.878326+0000", 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152249 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.878871+0000", 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054447700000000003 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 94 entries 57'17 (0'0) error 2:67cae2ed:::rbd_mirroring:head by client.5328.0:89 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'15)", 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.141309+0000", 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.681146257, 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072612700000000002, 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.141309+0000", 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.141309+0000", 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.141311+0000", 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.73e-06 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.141330+0000", 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.91e-05 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.141375+0000", 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4246000000000003e-05 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.141400+0000", 2024-09-21T23:26:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5643000000000002e-05 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.141563+0000", 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000162759 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.142036+0000", 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00047264900000000001 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 95 entries 57'18 (0'0) error 2:702f8319:::rbd_data.138b5612b65e.0000000000000000:head by client.5373.0:71 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'16)", 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.396629+0000", 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.4258272390000002, 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00067013700000000001, 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.396629+0000", 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.396629+0000", 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.396629+0000", 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5000000000000001e-07 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.396641+0000", 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1289e-05 2024-09-21T23:26:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.396668+0000", 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7237e-05 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.396681+0000", 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3339000000000001e-05 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.396781+0000", 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9315999999999996e-05 2024-09-21T23:26:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.397299+0000", 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000518006 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 97 entries 57'19 (0'0) error 2:7692ab5b:::rbd_data.138b5612b65e.0000000000000002:head by client.5373.0:73 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'16)", 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.397059+0000", 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.4253965400000004, 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00070992699999999995, 2024-09-21T23:26:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.397059+0000", 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.397059+0000", 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.397060+0000", 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8999999999999996e-07 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.397068+0000", 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2369999999999992e-06 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.397086+0000", 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8108999999999999e-05 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.397108+0000", 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2161e-05 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.397184+0000", 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5094999999999998e-05 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.397769+0000", 2024-09-21T23:26:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058563500000000004 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 108 entries 57'25 (0'0) error 2:6adbebdb:::rbd_data.138b5612b65e.000000000000002a:head by client.5373.0:114 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'23)", 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.413638+0000", 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.4088182079999996, 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00061368999999999996, 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413638+0000", 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413638+0000", 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413638+0000", 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1600000000000001e-07 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413646+0000", 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8839999999999994e-06 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413667+0000", 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1319e-05 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413683+0000", 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5216e-05 2024-09-21T23:26:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413770+0000", 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7342999999999994e-05 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.414251+0000", 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00048121200000000001 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 112 entries 57'26 (0'0) error 2:6eddcb1c:::rbd_data.138b5612b65e.000000000000003a:head by client.5373.0:130 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'24)", 2024-09-21T23:26:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.418863+0000", 2024-09-21T23:26:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.4035926090000004, 2024-09-21T23:26:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060325299999999999, 2024-09-21T23:26:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418863+0000", 2024-09-21T23:26:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418863+0000", 2024-09-21T23:26:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418864+0000", 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7800000000000001e-07 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418873+0000", 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8920000000000006e-06 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418906+0000", 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3198999999999998e-05 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418923+0000", 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6867e-05 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.419019+0000", 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6692000000000002e-05 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.419466+0000", 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00044692499999999998 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 57/42 rep_tid 114 entries 57'28 (0'0) error 2:662228e9:::journal_data.2.138b5612b65e.0:head by client.5373.0:149 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 57'26)", 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.432614+0000", 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.3898422400000001, 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00068386900000000004, 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.432614+0000", 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.432614+0000", 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.432615+0000", 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0249999999999999e-06 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.432623+0000", 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9699999999999999e-06 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.432645+0000", 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2823000000000002e-05 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.432673+0000", 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7413999999999999e-05 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.432773+0000", 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100608 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.433297+0000", 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052402899999999997 2024-09-21T23:26:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 59/42 rep_tid 191 entries 59'76 (0'0) error 2:7b1aaeb1:::rbd_data.15994fe581e3.0000000000000000:head by client.5580.0:71 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 59'74)", 2024-09-21T23:26:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.344723+0000", 2024-09-21T23:26:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 1.4777329830000001, 2024-09-21T23:26:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00064220200000000003, 2024-09-21T23:26:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.344723+0000", 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.344723+0000", 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.344724+0000", 2024-09-21T23:26:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3300000000000001e-07 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.344732+0000", 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8939999999999999e-06 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.344752+0000", 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9372999999999999e-05 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.344762+0000", 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0321e-05 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.344849+0000", 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6921000000000003e-05 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.345365+0000", 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00051586000000000004 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 59/42 rep_tid 202 entries 59'81 (0'0) error 2:792d34e0:::rbd_data.15994fe581e3.0000000000000020:head by client.5580.0:103 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 59'79)", 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.360090+0000", 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 1.4623655709999999, 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00069993300000000002, 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360090+0000", 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360090+0000", 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360091+0000", 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0800000000000004e-07 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360100+0000", 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5489999999999995e-06 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360121+0000", 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0551e-05 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360138+0000", 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7024000000000002e-05 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360222+0000", 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3807000000000006e-05 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360790+0000", 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000568294 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 59/42 rep_tid 208 entries 59'85 (0'0) error 2:718ee709:::journal_data.2.15994fe581e3.3:head by client.5580.0:151 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 59'83)", 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.565562+0000", 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 1.256894121, 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060904299999999998, 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.565562+0000", 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.565562+0000", 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.565563+0000", 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6899999999999996e-07 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.565577+0000", 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4718e-05 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.565601+0000", 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.391e-05 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.565630+0000", 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8435000000000001e-05 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.565710+0000", 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0159000000000003e-05 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.566171+0000", 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00046085199999999998 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "pg_update_log_missing(2.6 epoch 59/42 rep_tid 209 entries 59'86 (0'0) error 2:67cae2ed:::rbd_mirroring:head by client.5580.0:157 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 59'84)", 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.572459+0000", 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 1.249996267, 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00074002300000000003, 2024-09-21T23:26:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.572459+0000", 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.572459+0000", 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.572461+0000", 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.544e-06 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.572477+0000", 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6206e-05 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.572517+0000", 2024-09-21T23:26:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9579999999999997e-05 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.572541+0000", 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3790000000000001e-05 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.572711+0000", 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017020199999999999 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.573200+0000", 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00048870100000000002 2024-09-21T23:26:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:50.902 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:50.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T23:26:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is set 2024-09-21T23:26:51.011 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:51.011 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:51.011 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:51.011 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:51.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T23:26:51.127 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:51.127 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:51.127 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:51.127 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:51.127 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:51.136 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T23:26:51.244 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 83.079513320999993, 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0080986310000000002, 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:51.245 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163703+0000", 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163709+0000", 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163727+0000", 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7552000000000001e-05 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163744+0000", 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7541999999999999e-05 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.247 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.163757+0000", 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2855e-05 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164069+0000", 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031179899999999997 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.164084+0000", 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5265e-05 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167127+0000", 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030427029999999999 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167317+0000", 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000190006 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.167465+0000", 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014824 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.168309+0000", 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084350899999999999 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171730+0000", 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034217620000000001 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171770+0000", 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0074999999999998e-05 2024-09-21T23:26:51.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:28.171802+0000", 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1563999999999998e-05 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4244.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.074825000999994, 2024-09-21T23:26:51.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006147263, 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4244", 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/706923113", 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168392+0000", 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168394+0000", 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.956e-06 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168404+0000", 2024-09-21T23:26:51.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0463e-05 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168423+0000", 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8715999999999998e-05 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168432+0000", 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.516e-06 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168529+0000", 2024-09-21T23:26:51.254 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7073999999999999e-05 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.168534+0000", 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.831e-06 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170224+0000", 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016893590000000001 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170332+0000", 2024-09-21T23:26:51.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108346 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1", 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.170445+0000", 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112642 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.171210+0000", 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076543800000000001 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174469+0000", 2024-09-21T23:26:51.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003259086 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174506+0000", 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6465999999999999e-05 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:29.174539+0000", 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3370000000000001e-05 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.257 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5259.0:85 2.3 2:cfc8b22f:::rbd_data.13013c9ae692.000000000000000e:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:44.998344+0000", 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.2448727159999997, 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0056642680000000001, 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5259", 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 85 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.998344+0000", 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.998344+0000", 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.998345+0000", 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0400000000000005e-07 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.998349+0000", 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0260000000000001e-06 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.998360+0000", 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0857e-05 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.998365+0000", 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2870000000000004e-06 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.260 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.998401+0000", 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5713000000000002e-05 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:44.998470+0000", 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9160999999999997e-05 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.004008+0000", 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0055384199999999996 2024-09-21T23:26:51.261 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5259.0:90 2.2 2:4dd0ab61:::rbd_data.13013c9ae692.0000000000000013:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.001109+0000", 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.2421078229999996, 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0053292799999999996, 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5259", 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 90 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.001109+0000", 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.001109+0000", 2024-09-21T23:26:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.001110+0000", 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3099999999999997e-07 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.001114+0000", 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2899999999999996e-06 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.001125+0000", 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0718e-05 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.001130+0000", 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6149999999999996e-06 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.001186+0000", 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5968999999999999e-05 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.265 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.001329+0000", 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014257500000000001 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.006438+0000", 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005109382 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5259.0:96 2.2 2:4a2d9a29:::rbd_data.13013c9ae692.0000000000000019:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.004142+0000", 2024-09-21T23:26:51.266 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.2390746090000002, 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051255789999999999, 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5259", 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 96 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.004142+0000", 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.004142+0000", 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.004143+0000", 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7199999999999998e-07 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.004147+0000", 2024-09-21T23:26:51.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0280000000000002e-06 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.004157+0000", 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9110000000000005e-06 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.004163+0000", 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5270000000000004e-06 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.004214+0000", 2024-09-21T23:26:51.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0964999999999997e-05 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.004344+0000", 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000129214 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.009268+0000", 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049241620000000002 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5259.0:101 2.2 2:58bc8c56:::rbd_data.13013c9ae692.000000000000001e:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.009173+0000", 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.2340436190000004, 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0057681199999999998, 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5259", 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 101 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.009173+0000", 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.009173+0000", 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.009174+0000", 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4600000000000004e-07 2024-09-21T23:26:51.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.009177+0000", 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5039999999999998e-06 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.009185+0000", 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3590000000000002e-06 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.009189+0000", 2024-09-21T23:26:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2590000000000002e-06 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.009301+0000", 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011244 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.009375+0000", 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4155999999999999e-05 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.014941+0000", 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0055657559999999998 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5313.0:150 2.2 2:482257ae:::journal_data.2.135878064453.2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+full_force+supports_pool_eio e57)", 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.865315+0000", 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 5.3779017280000003, 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051168280000000003, 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.275 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5313", 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2898988500", 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 150 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865315+0000", 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865315+0000", 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865315+0000", 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3500000000000002e-07 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865319+0000", 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6830000000000001e-06 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865326+0000", 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1879999999999996e-06 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865332+0000", 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0240000000000003e-06 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865823+0000", 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049067399999999997 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865949+0000", 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125888 2024-09-21T23:26:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.870432+0000", 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044829359999999999 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5373.0:88 2.2 2:563bf02f:::rbd_data.138b5612b65e.0000000000000010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.400840+0000", 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.8423769889999999, 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.004896872, 2024-09-21T23:26:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5373", 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 88 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400840+0000", 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400840+0000", 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400841+0000", 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2750000000000001e-06 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400846+0000", 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5040000000000002e-06 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400862+0000", 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5238999999999999e-05 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400870+0000", 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3210000000000005e-06 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400940+0000", 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0248999999999998e-05 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.401068+0000", 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000127465 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.405737+0000", 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046688190000000003 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5373.0:89 2.3 2:c8e563f2:::rbd_data.138b5612b65e.0000000000000011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.283 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.400888+0000", 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.8423282499999996, 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049708390000000003, 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5373", 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 89 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400888+0000", 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400888+0000", 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400889+0000", 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3900000000000005e-07 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400892+0000", 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.932e-06 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400900+0000", 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5539999999999998e-06 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400910+0000", 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6539999999999994e-06 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400962+0000", 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2184e-05 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.401105+0000", 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014303800000000001 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.405859+0000", 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047539380000000001 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5373.0:94 2.2 2:53e7c4ba:::rbd_data.138b5612b65e.0000000000000016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.403210+0000", 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.8400070270000004, 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051839429999999999, 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5373", 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 94 2024-09-21T23:26:51.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403210+0000", 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403210+0000", 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403210+0000", 2024-09-21T23:26:51.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8899999999999998e-07 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403215+0000", 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0400000000000003e-06 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403225+0000", 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0538e-05 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403234+0000", 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7430000000000008e-06 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403291+0000", 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6749999999999997e-05 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403429+0000", 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000138329 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408394+0000", 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.004964754 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5373.0:97 2.3 2:c9bf251e:::rbd_data.138b5612b65e.0000000000000019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.403321+0000", 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.8398957669999998, 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0052781920000000001, 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.292 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5373", 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 97 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403321+0000", 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403321+0000", 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403321+0000", 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4600000000000005e-07 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403325+0000", 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.281e-06 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403332+0000", 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1099999999999997e-06 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403336+0000", 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7129999999999999e-06 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403386+0000", 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0467999999999999e-05 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.403532+0000", 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014623500000000001 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408599+0000", 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050668390000000001 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.296 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5373.0:105 2.2 2:4d58ba45:::rbd_data.138b5612b65e.0000000000000021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.408633+0000", 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.8345838069999996, 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0048088439999999996, 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5373", 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 105 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408633+0000", 2024-09-21T23:26:51.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408633+0000", 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408634+0000", 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2600000000000001e-07 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408645+0000", 2024-09-21T23:26:51.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1194e-05 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408659+0000", 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3601000000000001e-05 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408666+0000", 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0910000000000004e-06 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408728+0000", 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2558999999999995e-05 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408801+0000", 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2967999999999997e-05 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413442+0000", 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046405049999999996 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5373.0:106 2.2 2:5c8b9309:::rbd_data.138b5612b65e.0000000000000022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.408687+0000", 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.8345300340000001, 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049139489999999999, 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5373", 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 106 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408687+0000", 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408687+0000", 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408687+0000", 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.51e-07 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408691+0000", 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6279999999999998e-06 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408701+0000", 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.04e-05 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.408707+0000", 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2230000000000004e-06 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.409095+0000", 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000387952 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.409233+0000", 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013761200000000001 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413601+0000", 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043675830000000004 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.304 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5373.0:121 2.2 2:4ee67b53:::rbd_data.138b5612b65e.0000000000000031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.413898+0000", 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.8293187990000002, 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045905599999999996, 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5373", 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 121 2024-09-21T23:26:51.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413898+0000", 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413898+0000", 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413899+0000", 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7400000000000002e-07 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413926+0000", 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7656000000000001e-05 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413939+0000", 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2982e-05 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.413947+0000", 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7000000000000008e-06 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.414003+0000", 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5686999999999997e-05 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.414121+0000", 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011851200000000001 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418488+0000", 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043671489999999999 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5373.0:125 2.2 2:4cf829bc:::rbd_data.138b5612b65e.0000000000000035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.415965+0000", 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.827252101, 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051582950000000002, 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.309 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5373", 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.415965+0000", 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.415965+0000", 2024-09-21T23:26:51.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.415965+0000", 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2600000000000002e-07 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.415969+0000", 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4740000000000001e-06 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.415979+0000", 2024-09-21T23:26:51.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7599999999999997e-06 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.415986+0000", 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3900000000000004e-06 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.416022+0000", 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5598e-05 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.416091+0000", 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9227000000000007e-05 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.421123+0000", 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050321200000000002 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5373.0:134 2.2 2:4cf02c75:::rbd_data.138b5612b65e.000000000000003e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.418656+0000", 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.8245602840000004, 2024-09-21T23:26:51.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050637140000000004, 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5373", 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 134 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418656+0000", 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418656+0000", 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418657+0000", 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5499999999999998e-07 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418661+0000", 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7349999999999998e-06 2024-09-21T23:26:51.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418670+0000", 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6849999999999996e-06 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418676+0000", 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4840000000000003e-06 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418712+0000", 2024-09-21T23:26:51.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6591000000000003e-05 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.418789+0000", 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6427999999999994e-05 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.423720+0000", 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049312360000000003 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5373.0:156 2.3 2:cd7ac00d:::journal.138b5612b65e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.436242+0000", 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.8069745460000002, 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050692630000000001, 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5373", 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 156 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.436242+0000", 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.436242+0000", 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.436243+0000", 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6499999999999999e-07 2024-09-21T23:26:51.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.436245+0000", 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3479999999999998e-06 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.436252+0000", 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8390000000000001e-06 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.436257+0000", 2024-09-21T23:26:51.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4960000000000004e-06 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.436544+0000", 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000286784 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.436627+0000", 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3269000000000005e-05 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2", 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.436726+0000", 2024-09-21T23:26:51.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8444999999999997e-05 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.437238+0000", 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00051263000000000003 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.441261+0000", 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040225920000000002 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.441287+0000", 2024-09-21T23:26:51.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6100000000000001e-05 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.441311+0000", 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4295e-05 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5580.0:107 2.3 2:d6c86c89:::rbd_data.15994fe581e3.0000000000000024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.360162+0000", 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 1.8830546350000001, 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.004596022, 2024-09-21T23:26:51.323 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5580", 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 107 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360162+0000", 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360162+0000", 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360163+0000", 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0699999999999996e-07 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360167+0000", 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2289999999999997e-06 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360192+0000", 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4878000000000001e-05 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360199+0000", 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3950000000000003e-06 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360253+0000", 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3143999999999998e-05 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.360347+0000", 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4964999999999996e-05 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.364758+0000", 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044106040000000003 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.327 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5580.0:117 2.3 2:c05a0b12:::rbd_data.15994fe581e3.000000000000002e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.370315+0000", 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 1.8729013939999999, 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051745970000000004, 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5580", 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 117 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.370315+0000", 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.370315+0000", 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.370316+0000", 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1200000000000001e-07 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.370321+0000", 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.476e-06 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.370331+0000", 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0261000000000001e-05 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.370338+0000", 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1430000000000001e-06 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.370457+0000", 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119239 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.370579+0000", 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012211500000000001 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.375490+0000", 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049104509999999997 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.331 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5580.0:123 2.3 2:c5dd101a:::rbd_data.15994fe581e3.0000000000000034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.375614+0000", 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 1.867602558, 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050660760000000001, 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5580", 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 123 2024-09-21T23:26:51.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.375614+0000", 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.375614+0000", 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.375615+0000", 2024-09-21T23:26:51.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4000000000000001e-07 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.375618+0000", 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6629999999999999e-06 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.375629+0000", 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0380999999999999e-05 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.375642+0000", 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3138e-05 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.375690+0000", 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7895000000000001e-05 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.375811+0000", 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121222 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.380680+0000", 2024-09-21T23:26:51.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0048691369999999999 2024-09-21T23:26:51.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.336 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.336 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.336 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:51.337 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T23:26:51.444 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:51.445 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:51.445 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:51.445 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:51.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T23:26:51.560 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:51.560 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:51.560 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:51.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T23:26:51.677 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5259.0:72 2.7 2:ed01631b:::rbd_data.13013c9ae692.0000000000000001:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.194990+0000", 2024-09-21T23:26:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.4805376849999998, 2024-09-21T23:26:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030280699999999999, 2024-09-21T23:26:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5259", 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 72 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.194990+0000", 2024-09-21T23:26:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.194990+0000", 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.194991+0000", 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.265e-06 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.194995+0000", 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2690000000000001e-06 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195003+0000", 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2029999999999992e-06 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195012+0000", 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0890000000000006e-06 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195094+0000", 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1678e-05 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195102+0000", 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3229999999999998e-06 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.196467+0000", 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001365029 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.196545+0000", 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7867999999999994e-05 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.198018+0000", 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001472346 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5259.0:82 2.7 2:f616381f:::rbd_data.13013c9ae692.000000000000000b:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.195028+0000", 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.480499096, 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031310190000000001, 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5259", 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 82 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195028+0000", 2024-09-21T23:26:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195028+0000", 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195029+0000", 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1600000000000002e-07 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195032+0000", 2024-09-21T23:26:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0460000000000001e-06 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195040+0000", 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5129999999999996e-06 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195045+0000", 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8250000000000004e-06 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195170+0000", 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012515399999999999 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting_for_map not empty", 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195173+0000", 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.292e-06 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.196814+0000", 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016403399999999999 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.196903+0000", 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9606999999999998e-05 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.198159+0000", 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001255826 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5259.0:99 2.7 2:fbf6858b:::rbd_data.13013c9ae692.000000000000001c:head [delete] snapc 3=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.195070+0000", 2024-09-21T23:26:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.4804572250000003, 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031762930000000002, 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5259", 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/3853830039", 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 99 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195070+0000", 2024-09-21T23:26:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195070+0000", 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195071+0000", 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4999999999999998e-07 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195085+0000", 2024-09-21T23:26:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4669e-05 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195092+0000", 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8480000000000003e-06 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195096+0000", 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6959999999999998e-06 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195221+0000", 2024-09-21T23:26:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125463 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting_for_map not empty", 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.195224+0000", 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1190000000000001e-06 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.197159+0000", 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019347240000000001 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.197300+0000", 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000140672 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.198246+0000", 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00094665199999999998 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4946.0:89 2.7 2:e08d089a:::rbd_data.1286a02efebc.0000000000000012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.309988+0000", 2024-09-21T23:26:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.3655389109999998, 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001446307, 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4946", 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/3823599977", 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 89 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.309988+0000", 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.309988+0000", 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.309989+0000", 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1799999999999995e-07 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.309993+0000", 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6200000000000001e-06 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.310002+0000", 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2099999999999999e-06 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.310009+0000", 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8650000000000003e-06 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.310485+0000", 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00047596999999999998 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.310636+0000", 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000151812 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.311435+0000", 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00079821199999999997 2024-09-21T23:26:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5313.0:73 2.7 2:f4635dab:::rbd_data.135878064453.0000000000000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.649153+0000", 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.0263740930000003, 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001433047, 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5313", 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2898988500", 2024-09-21T23:26:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 73 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649153+0000", 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649153+0000", 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649154+0000", 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5099999999999999e-07 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649162+0000", 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.585e-06 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649170+0000", 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7249999999999997e-06 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649174+0000", 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0999999999999997e-06 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649216+0000", 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2104999999999998e-05 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649334+0000", 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117936 2024-09-21T23:26:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.650586+0000", 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001251945 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5313.0:79 2.7 2:ffdc897c:::rbd_data.135878064453.0000000000000008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.649258+0000", 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.0262694469999998, 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019943230000000001, 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5313", 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2898988500", 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 79 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649258+0000", 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649258+0000", 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649259+0000", 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.015e-06 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649264+0000", 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7709999999999997e-06 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649281+0000", 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7719000000000001e-05 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649290+0000", 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2190000000000004e-06 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649635+0000", 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00034515199999999999 2024-09-21T23:26:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649771+0000", 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000136303 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.651252+0000", 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001481144 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5313.0:80 2.7 2:feaf96ca:::rbd_data.135878064453.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.649309+0000", 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.0262183379999996, 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022035119999999999, 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5313", 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2898988500", 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 80 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649309+0000", 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649309+0000", 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649310+0000", 2024-09-21T23:26:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3200000000000005e-07 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649313+0000", 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1420000000000002e-06 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649321+0000", 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2830000000000001e-06 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.649327+0000", 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2589999999999997e-06 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.650062+0000", 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00073504899999999999 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.650186+0000", 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012348 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.651512+0000", 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013266669999999999 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5313.0:94 2.7 2:e80b2336:::rbd_data.135878064453.0000000000000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.653030+0000", 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.022497703, 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015602249999999999, 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5313", 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2898988500", 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 94 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.653030+0000", 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.653030+0000", 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.653030+0000", 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5899999999999995e-07 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.653035+0000", 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.515e-06 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.653045+0000", 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8139999999999995e-06 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.653056+0000", 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0852999999999999e-05 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.653288+0000", 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023282199999999999 2024-09-21T23:26:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.653408+0000", 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119449 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.654590+0000", 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011819129999999999 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5313.0:124 2.7 2:e011a8cd:::rbd_data.135878064453.0000000000000035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.660891+0000", 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.0146359690000004, 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014394659999999999, 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5313", 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2898988500", 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.660891+0000", 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.660891+0000", 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.660892+0000", 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.09e-07 2024-09-21T23:26:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.660896+0000", 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3050000000000003e-06 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.660906+0000", 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6439999999999995e-06 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.660914+0000", 2024-09-21T23:26:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8280000000000003e-06 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.661143+0000", 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022943099999999999 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.661269+0000", 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125334 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.662331+0000", 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001062115 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5313.0:125 2.7 2:f22d5260:::rbd_data.135878064453.0000000000000036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.661087+0000", 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 6.0144399330000002, 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019650570000000001, 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5313", 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2898988500", 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.661087+0000", 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.661087+0000", 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.661088+0000", 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3900000000000004e-07 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.661093+0000", 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8810000000000004e-06 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.661105+0000", 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1649e-05 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.661110+0000", 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3319999999999999e-06 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.661597+0000", 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00048651199999999998 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.661686+0000", 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9345000000000006e-05 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.663052+0000", 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001366499 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5313.0:153 2.7 2:ea45bf02:::journal_data.2.135878064453.5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+full_force+supports_pool_eio e57)", 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:45.865304+0000", 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 5.810223744, 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00161382, 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5313", 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2898988500", 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 153 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865304+0000", 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865304+0000", 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865304+0000", 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7100000000000002e-07 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865308+0000", 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.579e-06 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865316+0000", 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2009999999999999e-06 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865322+0000", 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2720000000000003e-06 2024-09-21T23:26:51.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865809+0000", 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00048639700000000001 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.865926+0000", 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011764000000000001 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:45.866917+0000", 2024-09-21T23:26:51.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099116000000000004 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5373.0:87 2.7 2:f5b7cc77:::rbd_data.138b5612b65e.000000000000000f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e57)", 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:46.400445+0000", 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 5.2750822560000001, 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014965989999999999, 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.728 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5373", 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/2679851225", 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400445+0000", 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.729 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400445+0000", 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400445+0000", 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3200000000000001e-07 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400447+0000", 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.807e-06 2024-09-21T23:26:51.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400452+0000", 2024-09-21T23:26:51.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0980000000000001e-06 2024-09-21T23:26:51.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400455+0000", 2024-09-21T23:26:51.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0920000000000002e-06 2024-09-21T23:26:51.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400773+0000", 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031750799999999998 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.400889+0000", 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116527 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:46.401942+0000", 2024-09-21T23:26:51.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010522350000000001 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5580.0:77 2.7 2:f6ce3371:::rbd_data.15994fe581e3.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.546708+0000", 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 2.1288195889999999, 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020188390000000001, 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.733 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5580", 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 77 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546708+0000", 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546708+0000", 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546710+0000", 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1600000000000001e-06 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546720+0000", 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7100000000000002e-06 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546743+0000", 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3144000000000001e-05 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546765+0000", 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1985999999999999e-05 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546893+0000", 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012878299999999999 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547081+0000", 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018760099999999999 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.548727+0000", 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016454550000000001 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.737 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5580.0:81 2.7 2:fd47543d:::rbd_data.15994fe581e3.000000000000000a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.546812+0000", 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 2.12871531, 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024158069999999999, 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5580", 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 81 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546812+0000", 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546812+0000", 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546813+0000", 2024-09-21T23:26:51.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.477e-06 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546821+0000", 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4669999999999998e-06 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546839+0000", 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8193e-05 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546856+0000", 2024-09-21T23:26:51.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6716999999999999e-05 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547484+0000", 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00062801700000000003 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547561+0000", 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7001000000000004e-05 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.549228+0000", 2024-09-21T23:26:51.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001666935 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5580.0:92 2.7 2:fc8aecc4:::rbd_data.15994fe581e3.0000000000000015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.546904+0000", 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 2.128623277, 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024797560000000001, 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.742 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5580", 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 92 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546904+0000", 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546904+0000", 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546905+0000", 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2839999999999999e-06 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546912+0000", 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8000000000000001e-06 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546934+0000", 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1542000000000001e-05 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546949+0000", 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5526999999999999e-05 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547877+0000", 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000928055 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547983+0000", 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000105818 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.549384+0000", 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00140073 2024-09-21T23:26:51.745 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5580.0:100 2.7 2:eb204ed3:::rbd_data.15994fe581e3.000000000000001d:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.546973+0000", 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 2.128554737, 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0025123910000000001, 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.746 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5580", 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 100 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546973+0000", 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546973+0000", 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546974+0000", 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.206e-06 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.546980+0000", 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7740000000000001e-06 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547012+0000", 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2119999999999997e-05 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547026+0000", 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4633e-05 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.548247+0000", 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012210560000000001 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.548362+0000", 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000114473 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.549485+0000", 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011231290000000001 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5580.0:102 2.7 2:f527d810:::rbd_data.15994fe581e3.000000000000001f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.547048+0000", 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 2.1284789110000002, 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032399099999999999, 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.750 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5580", 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 102 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547048+0000", 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547048+0000", 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547050+0000", 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.159e-06 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547066+0000", 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6344000000000001e-05 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547087+0000", 2024-09-21T23:26:51.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0656e-05 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547102+0000", 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5440000000000001e-05 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.548786+0000", 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016837289999999999 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.548889+0000", 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102859 2024-09-21T23:26:51.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.550288+0000", 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013997230000000001 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5580.0:109 2.7 2:e3797189:::rbd_data.15994fe581e3.0000000000000026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.547121+0000", 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 2.1284063450000001, 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037177070000000002, 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.754 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5580", 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 109 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547121+0000", 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547121+0000", 2024-09-21T23:26:51.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547122+0000", 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.217e-06 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547128+0000", 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5860000000000004e-06 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547146+0000", 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8403e-05 2024-09-21T23:26:51.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547161+0000", 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.489e-05 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.549525+0000", 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023636159999999998 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.549603+0000", 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8053000000000006e-05 2024-09-21T23:26:51.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.550839+0000", 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012359420000000001 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5580.0:115 2.7 2:e8f12ceb:::rbd_data.15994fe581e3.000000000000002c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.547179+0000", 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 2.128348216, 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0039502290000000004, 2024-09-21T23:26:51.758 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5580", 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 115 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547179+0000", 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.759 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547179+0000", 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547180+0000", 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0729999999999999e-06 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547189+0000", 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1470000000000007e-06 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547210+0000", 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0706000000000001e-05 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547217+0000", 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1359999999999999e-06 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.549856+0000", 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026389880000000001 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.549948+0000", 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1828000000000002e-05 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.551129+0000", 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001181351 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.5580.0:134 2.7 2:e8689a91:::rbd_data.15994fe581e3.000000000000003f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e59)", 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:26:49.547226+0000", 2024-09-21T23:26:51.762 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 2.1283017489999998, 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040928509999999998, 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.5580", 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.31:0/4002515302", 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 134 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547226+0000", 2024-09-21T23:26:51.763 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547226+0000", 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547226+0000", 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8599999999999999e-07 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547228+0000", 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2079999999999999e-06 2024-09-21T23:26:51.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547235+0000", 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1879999999999996e-06 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.547242+0000", 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0110000000000003e-06 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.550349+0000", 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031064349999999998 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.550477+0000", 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012795600000000001 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:26:49.551318+0000", 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084166700000000004 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:51.766 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:51.767 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T23:26:51.875 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:51.875 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:51.875 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-21T23:26:51.876 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:51.884 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T23:26:51.991 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:51.991 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-21T23:26:51.991 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-09-21T23:26:51.991 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-09-21T23:26:51.991 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:51.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T23:26:52.108 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.940025629000004, 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090066050000000009, 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 37 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167006+0000", 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7099999999999998e-07 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167009+0000", 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.92e-06 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167013+0000", 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.167017+0000", 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7280000000000001e-06 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170778+0000", 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037612399999999999 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170831+0000", 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2694000000000001e-05 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.170970+0000", 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000138966 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.171555+0000", 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058538399999999999 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175872+0000", 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043172389999999996 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.175969+0000", 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6844999999999999e-05 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.176012+0000", 2024-09-21T23:26:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3356999999999998e-05 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.913920793000003, 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069023069999999999, 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193111+0000", 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0999999999999999e-07 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193115+0000", 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5700000000000001e-06 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193129+0000", 2024-09-21T23:26:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4124000000000001e-05 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.193133+0000", 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0550000000000001e-06 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194144+0000", 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001010735 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194231+0000", 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7665999999999997e-05 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194357+0000", 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125353 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.194844+0000", 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000487305 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199964+0000", 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051204859999999996 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.199992+0000", 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7902000000000002e-05 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.200013+0000", 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0601000000000001e-05 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.892271719000007, 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0067494620000000003, 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 73 2024-09-21T23:26:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214760+0000", 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4900000000000001e-07 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214818+0000", 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7754000000000003e-05 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214842+0000", 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4552000000000001e-05 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214846+0000", 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9129999999999996e-06 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215813+0000", 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00096695899999999998 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.215904+0000", 2024-09-21T23:26:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0328000000000007e-05 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216020+0000", 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115838 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217018+0000", 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099836200000000003 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221461+0000", 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044435969999999997 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221487+0000", 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5806e-05 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221509+0000", 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1903999999999999e-05 2024-09-21T23:26:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.892177016000005, 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068139230000000004, 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 74 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214854+0000", 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214855+0000", 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-07 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214859+0000", 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1389999999999997e-06 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214871+0000", 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1698999999999999e-05 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.214887+0000", 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6061000000000001e-05 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216359+0000", 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014720779999999999 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216439+0000", 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0191999999999994e-05 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.216579+0000", 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014029099999999999 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.217129+0000", 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054932099999999997 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221559+0000", 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044299710000000004 2024-09-21T23:26:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221627+0000", 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8631000000000001e-05 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.221668+0000", 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0988000000000001e-05 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.875500829999993, 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0077112509999999997, 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 86 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231531+0000", 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5199999999999998e-07 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231533+0000", 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3879999999999998e-06 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231537+0000", 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0010000000000003e-06 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231540+0000", 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0489999999999999e-06 2024-09-21T23:26:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233890+0000", 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023500249999999999 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.233937+0000", 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6400999999999998e-05 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234015+0000", 2024-09-21T23:26:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8343000000000005e-05 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235675+0000", 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001660124 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239138+0000", 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034624109999999999 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239200+0000", 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2698000000000002e-05 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.239242+0000", 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1458999999999997e-05 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.875484560999993, 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010200097999999999, 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231547+0000", 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2700000000000001e-07 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231588+0000", 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0985000000000001e-05 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231593+0000", 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1059999999999999e-06 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231606+0000", 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2343e-05 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234406+0000", 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027999850000000001 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234489+0000", 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3051000000000002e-05 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.234631+0000", 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285799999999999 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235679+0000", 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001047517 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241644+0000", 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059652129999999996 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241704+0000", 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9732999999999999e-05 2024-09-21T23:26:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241747+0000", 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-05 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.875418722000006, 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010246534, 2024-09-21T23:26:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 88 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231613+0000", 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0199999999999998e-07 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231615+0000", 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3439999999999999e-06 2024-09-21T23:26:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231619+0000", 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0219999999999998e-06 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.231622+0000", 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6989999999999999e-06 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235337+0000", 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037150030000000001 2024-09-21T23:26:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235385+0000", 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7861999999999997e-05 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235458+0000", 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2596999999999996e-05 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.235978+0000", 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00052013600000000004 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241799+0000", 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058215289999999998 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241836+0000", 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6282999999999998e-05 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.241859+0000", 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3757e-05 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.718027539000005, 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072206969999999999, 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 112 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389004+0000", 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.72e-07 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389069+0000", 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4875000000000005e-05 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389079+0000", 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0135e-05 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389085+0000", 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394006+0000", 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049207920000000002 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394052+0000", 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5810999999999999e-05 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394139+0000", 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7130000000000001e-05 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395407+0000", 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012681260000000001 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396139+0000", 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000731703 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396194+0000", 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5309999999999997e-05 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396225+0000", 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0589000000000001e-05 2024-09-21T23:26:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.717920582000005, 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086923490000000003, 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 113 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389111+0000", 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389116+0000", 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.245e-06 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389126+0000", 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0295000000000001e-05 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389132+0000", 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0889999999999996e-06 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394548+0000", 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005415876 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394642+0000", 2024-09-21T23:26:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4531999999999995e-05 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.394714+0000", 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1753000000000002e-05 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396233+0000", 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015185299999999999 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397749+0000", 2024-09-21T23:26:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015157770000000001 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397779+0000", 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0561000000000001e-05 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397803+0000", 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4116000000000001e-05 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.717888513000005, 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.01139824, 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 114 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389143+0000", 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.01e-07 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389146+0000", 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5239999999999999e-06 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389150+0000", 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3989999999999997e-06 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389153+0000", 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.038e-06 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395026+0000", 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0058731720000000003 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395071+0000", 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4388999999999998e-05 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395141+0000", 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0053000000000002e-05 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397196+0000", 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020555899999999999 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400494+0000", 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032972679999999999 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400520+0000", 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6423000000000001e-05 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400541+0000", 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0982999999999999e-05 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.717869833999998, 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011537824, 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 115 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389162+0000", 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4900000000000001e-07 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389197+0000", 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5204999999999999e-05 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389203+0000", 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0979999999999999e-06 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389207+0000", 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6600000000000001e-06 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395443+0000", 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062358099999999996 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395525+0000", 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1855999999999997e-05 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.395669+0000", 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014472399999999999 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397204+0000", 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001534817 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400599+0000", 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033944499999999998 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400655+0000", 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6647000000000002e-05 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400699+0000", 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4208000000000001e-05 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.717818382000004, 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011571580999999999, 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 116 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389213+0000", 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3099999999999999e-07 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389216+0000", 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4260000000000002e-06 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389223+0000", 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389225+0000", 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7760000000000002e-06 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396256+0000", 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070309020000000003 2024-09-21T23:26:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396348+0000", 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1780000000000006e-05 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396481+0000", 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000133101 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397208+0000", 2024-09-21T23:26:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072716199999999995 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400737+0000", 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035290220000000001 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400763+0000", 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5219999999999999e-05 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400785+0000", 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2045e-05 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.717801699000006, 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011687858000000001, 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 117 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389230+0000", 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3699999999999999e-07 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389232+0000", 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1789999999999998e-06 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389238+0000", 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209999999999999e-06 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.389241+0000", 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.557e-06 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396792+0000", 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075506899999999997 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396838+0000", 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5639000000000001e-05 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.396958+0000", 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119892 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.397458+0000", 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049987600000000001 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400824+0000", 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033662560000000002 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400880+0000", 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5779999999999998e-05 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.400918+0000", 2024-09-21T23:26:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8031000000000002e-05 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.705783225000005, 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006886924, 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 120 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401248+0000", 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401249+0000", 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.13e-07 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401286+0000", 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6773999999999997e-05 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401295+0000", 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9039999999999998e-06 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401299+0000", 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3250000000000001e-06 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402183+0000", 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000883758 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402264+0000", 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1080000000000003e-05 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402401+0000", 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013719700000000001 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403430+0000", 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010290119999999999 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408045+0000", 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046146659999999999 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408095+0000", 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0898e-05 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408135+0000", 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9697000000000003e-05 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:121 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.705724751000005, 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069210920000000002, 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 121 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401307+0000", 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3200000000000001e-07 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401341+0000", 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4236999999999998e-05 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401346+0000", 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.04e-06 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.401350+0000", 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6399999999999999e-06 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402733+0000", 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013825790000000001 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402816+0000", 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3461000000000006e-05 2024-09-21T23:26:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.402941+0000", 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012518300000000001 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.403533+0000", 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000592071 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408174+0000", 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046404209999999996 2024-09-21T23:26:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408204+0000", 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0147000000000001e-05 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.408228+0000", 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3981e-05 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.269581853999995, 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064924400000000004, 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 139 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837450+0000", 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0800000000000001e-07 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837452+0000", 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7209999999999999e-06 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837455+0000", 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2279999999999998e-06 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837458+0000", 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8200000000000001e-06 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842187+0000", 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047298460000000002 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842233+0000", 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6056000000000001e-05 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842304+0000", 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0862e-05 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843291+0000", 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00098632100000000007 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843894+0000", 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060369199999999997 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843922+0000", 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7429000000000001e-05 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843942+0000", 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0256999999999999e-05 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.269569911000005, 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066155550000000004, 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 140 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837462+0000", 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1e-07 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837517+0000", 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5034000000000002e-05 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837522+0000", 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8520000000000003e-06 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837525+0000", 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4139999999999998e-06 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842562+0000", 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050366279999999996 2024-09-21T23:26:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842615+0000", 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3675999999999999e-05 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.842715+0000", 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9990999999999999e-05 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843873+0000", 2024-09-21T23:26:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011573 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844026+0000", 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015368900000000001 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844058+0000", 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1449999999999999e-05 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844077+0000", 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9310999999999999e-05 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.269497717999997, 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068324850000000001, 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 141 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837534+0000", 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.22e-07 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837539+0000", 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.023e-06 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837543+0000", 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6490000000000001e-06 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.837546+0000", 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.382e-06 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843479+0000", 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059328189999999998 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843519+0000", 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0071000000000003e-05 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.843600+0000", 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1376000000000002e-05 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844137+0000", 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053640299999999997 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844319+0000", 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000181893 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844346+0000", 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.781e-05 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:21.844366+0000", 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9737000000000001e-05 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 89.654164023000007, 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006594145, 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 158 2024-09-21T23:26:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452867+0000", 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452868+0000", 2024-09-21T23:26:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9500000000000003e-07 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452938+0000", 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9598999999999996e-05 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452950+0000", 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2971e-05 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452960+0000", 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.448e-06 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456217+0000", 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003257378 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456557+0000", 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033989399999999998 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.456703+0000", 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000145599 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459339+0000", 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026365529999999998 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459386+0000", 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6640000000000001e-05 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459428+0000", 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2148999999999999e-05 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459462+0000", 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3419e-05 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T23:26:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 89.654059649000004, 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0065672760000000004, 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.4201", 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/125215748", 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 159 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452972+0000", 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5400000000000001e-07 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452977+0000", 2024-09-21T23:26:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.301e-06 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452984+0000", 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1840000000000002e-06 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.452989+0000", 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5219999999999998e-06 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-09-21T23:26:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457324+0000", 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043348780000000003 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457416+0000", 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1824999999999996e-05 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0", 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.457574+0000", 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015810999999999999 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459376+0000", 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001801649 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459467+0000", 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1367999999999997e-05 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459507+0000", 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9576000000000002e-05 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-21T23:26:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-09-21T23:26:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-09-21T23:25:22.459539+0000", 2024-09-21T23:26:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2308999999999998e-05 2024-09-21T23:26:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-21T23:26:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-21T23:26:52.225 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-21T23:26:52.285 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T23:26:52.415 INFO:teuthology.orchestra.run.smithi031.stdout:mon.a: {} 2024-09-21T23:26:52.416 INFO:teuthology.orchestra.run.smithi031.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T23:26:52.425 INFO:teuthology.orchestra.run.smithi031.stdout:mon.b: {} 2024-09-21T23:26:52.425 INFO:teuthology.orchestra.run.smithi031.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T23:26:52.434 INFO:teuthology.orchestra.run.smithi031.stdout:mon.c: {} 2024-09-21T23:26:52.434 INFO:teuthology.orchestra.run.smithi031.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T23:26:52.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-21T23:26:52.921 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T23:26:52.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T23:26:52.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-21T23:26:53.850 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T23:26:53.851 INFO:teuthology.orchestra.run.smithi031.stderr:noscrub is unset 2024-09-21T23:26:53.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T23:26:53.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-21T23:26:54.852 INFO:teuthology.orchestra.run.smithi031.stderr:nodeep-scrub is unset 2024-09-21T23:26:54.853 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T23:26:54.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-21T23:26:55.939 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T23:26:55.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-21T23:26:56.941 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T23:26:56.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-21T23:26:57.954 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T23:26:57.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-21T23:26:58.962 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T23:26:58.980 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-21T23:26:59.966 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T23:26:59.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-21T23:27:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T23:27:00.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-21T23:27:01.986 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T23:27:02.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-21T23:27:02.991 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T23:27:03.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-21T23:27:04.003 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T23:27:04.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-21T23:27:05.002 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T23:27:05.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-21T23:27:06.008 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T23:27:06.025 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-21T23:27:07.018 INFO:teuthology.orchestra.run.smithi031.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T23:27:07.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-21T23:27:08.027 INFO:teuthology.orchestra.run.smithi031.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-21T23:27:08.045 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T23:27:08.045 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T23:27:08.045 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T23:27:08.045 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T23:27:08.045 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T23:27:08.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T23:27:08.344 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:27:08.344 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":76,"fsid":"d51cc561-ddb8-41ef-b61c-7c6d698e4dd0","created":"2024-09-21T23:25:09.056809+0000","modified":"2024-09-21T23:27:08.020330+0000","last_up_change":"2024-09-21T23:25:20.126996+0000","last_in_change":"2024-09-21T23:25:11.226810+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:25:19.818212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T23:25:26.968096+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6817","nonce":2339966024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6819","nonce":2339966024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6823","nonce":2339966024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6821","nonce":2339966024}]},"public_addr":"172.21.15.31:6817/2339966024","cluster_addr":"172.21.15.31:6819/2339966024","heartbeat_back_addr":"172.21.15.31:6823/2339966024","heartbeat_front_addr":"172.21.15.31:6821/2339966024","state":["exists","up"]},{"osd":1,"uuid":"88ee06a3-9a6a-41e8-9327-a7a61304c03d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6825","nonce":2889649822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6827","nonce":2889649822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6831","nonce":2889649822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6829","nonce":2889649822}]},"public_addr":"172.21.15.31:6825/2889649822","cluster_addr":"172.21.15.31:6827/2889649822","heartbeat_back_addr":"172.21.15.31:6831/2889649822","heartbeat_front_addr":"172.21.15.31:6829/2889649822","state":["exists","up"]},{"osd":2,"uuid":"f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6809","nonce":864917651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6811","nonce":864917651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6815","nonce":864917651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6813","nonce":864917651}]},"public_addr":"172.21.15.31:6809/864917651","cluster_addr":"172.21.15.31:6811/864917651","heartbeat_back_addr":"172.21.15.31:6815/864917651","heartbeat_front_addr":"172.21.15.31:6813/864917651","state":["exists","up"]},{"osd":3,"uuid":"562e8c30-0622-4cb3-b5d1-f15b42a444bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6801","nonce":1713632981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6803","nonce":1713632981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6807","nonce":1713632981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6805","nonce":1713632981}]},"public_addr":"172.21.15.31:6801/1713632981","cluster_addr":"172.21.15.31:6803/1713632981","heartbeat_back_addr":"172.21.15.31:6807/1713632981","heartbeat_front_addr":"172.21.15.31:6805/1713632981","state":["exists","up"]},{"osd":4,"uuid":"7030ff68-4556-4ad5-abc9-be758f1caa85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6809","nonce":2660753894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6811","nonce":2660753894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6815","nonce":2660753894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6813","nonce":2660753894}]},"public_addr":"172.21.15.60:6809/2660753894","cluster_addr":"172.21.15.60:6811/2660753894","heartbeat_back_addr":"172.21.15.60:6815/2660753894","heartbeat_front_addr":"172.21.15.60:6813/2660753894","state":["exists","up"]},{"osd":5,"uuid":"7169a163-f067-4385-b734-f58a3ba9e89e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6825","nonce":2320687649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6827","nonce":2320687649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6831","nonce":2320687649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6829","nonce":2320687649}]},"public_addr":"172.21.15.60:6825/2320687649","cluster_addr":"172.21.15.60:6827/2320687649","heartbeat_back_addr":"172.21.15.60:6831/2320687649","heartbeat_front_addr":"172.21.15.60:6829/2320687649","state":["exists","up"]},{"osd":6,"uuid":"e511295d-2177-4dcb-a74b-b16b92534df6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6801","nonce":3490482162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6803","nonce":3490482162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6807","nonce":3490482162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6805","nonce":3490482162}]},"public_addr":"172.21.15.60:6801/3490482162","cluster_addr":"172.21.15.60:6803/3490482162","heartbeat_back_addr":"172.21.15.60:6807/3490482162","heartbeat_front_addr":"172.21.15.60:6805/3490482162","state":["exists","up"]},{"osd":7,"uuid":"544f8a91-ce82-4855-85eb-ea814eb3d058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6817","nonce":1315554966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6819","nonce":1315554966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6823","nonce":1315554966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6821","nonce":1315554966}]},"public_addr":"172.21.15.60:6817/1315554966","cluster_addr":"172.21.15.60:6819/1315554966","heartbeat_back_addr":"172.21.15.60:6823/1315554966","heartbeat_front_addr":"172.21.15.60:6821/1315554966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.514872+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.912129+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.745154+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:16.758871+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.287462+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:18.031636+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.055855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.524609+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[0,1]},{"pgid":"2.6","osds":[2,4]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:27:08.357 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T23:27:08.357 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T23:27:08.647 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:27:08.647 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":76,"fsid":"d51cc561-ddb8-41ef-b61c-7c6d698e4dd0","created":"2024-09-21T23:25:09.056809+0000","modified":"2024-09-21T23:27:08.020330+0000","last_up_change":"2024-09-21T23:25:20.126996+0000","last_in_change":"2024-09-21T23:25:11.226810+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:25:19.818212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T23:25:26.968096+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6817","nonce":2339966024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6819","nonce":2339966024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6823","nonce":2339966024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6821","nonce":2339966024}]},"public_addr":"172.21.15.31:6817/2339966024","cluster_addr":"172.21.15.31:6819/2339966024","heartbeat_back_addr":"172.21.15.31:6823/2339966024","heartbeat_front_addr":"172.21.15.31:6821/2339966024","state":["exists","up"]},{"osd":1,"uuid":"88ee06a3-9a6a-41e8-9327-a7a61304c03d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6825","nonce":2889649822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6827","nonce":2889649822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6831","nonce":2889649822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6829","nonce":2889649822}]},"public_addr":"172.21.15.31:6825/2889649822","cluster_addr":"172.21.15.31:6827/2889649822","heartbeat_back_addr":"172.21.15.31:6831/2889649822","heartbeat_front_addr":"172.21.15.31:6829/2889649822","state":["exists","up"]},{"osd":2,"uuid":"f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6809","nonce":864917651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6811","nonce":864917651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6815","nonce":864917651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6813","nonce":864917651}]},"public_addr":"172.21.15.31:6809/864917651","cluster_addr":"172.21.15.31:6811/864917651","heartbeat_back_addr":"172.21.15.31:6815/864917651","heartbeat_front_addr":"172.21.15.31:6813/864917651","state":["exists","up"]},{"osd":3,"uuid":"562e8c30-0622-4cb3-b5d1-f15b42a444bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6801","nonce":1713632981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6803","nonce":1713632981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6807","nonce":1713632981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6805","nonce":1713632981}]},"public_addr":"172.21.15.31:6801/1713632981","cluster_addr":"172.21.15.31:6803/1713632981","heartbeat_back_addr":"172.21.15.31:6807/1713632981","heartbeat_front_addr":"172.21.15.31:6805/1713632981","state":["exists","up"]},{"osd":4,"uuid":"7030ff68-4556-4ad5-abc9-be758f1caa85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6809","nonce":2660753894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6811","nonce":2660753894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6815","nonce":2660753894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6813","nonce":2660753894}]},"public_addr":"172.21.15.60:6809/2660753894","cluster_addr":"172.21.15.60:6811/2660753894","heartbeat_back_addr":"172.21.15.60:6815/2660753894","heartbeat_front_addr":"172.21.15.60:6813/2660753894","state":["exists","up"]},{"osd":5,"uuid":"7169a163-f067-4385-b734-f58a3ba9e89e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6825","nonce":2320687649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6827","nonce":2320687649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6831","nonce":2320687649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6829","nonce":2320687649}]},"public_addr":"172.21.15.60:6825/2320687649","cluster_addr":"172.21.15.60:6827/2320687649","heartbeat_back_addr":"172.21.15.60:6831/2320687649","heartbeat_front_addr":"172.21.15.60:6829/2320687649","state":["exists","up"]},{"osd":6,"uuid":"e511295d-2177-4dcb-a74b-b16b92534df6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6801","nonce":3490482162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6803","nonce":3490482162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6807","nonce":3490482162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6805","nonce":3490482162}]},"public_addr":"172.21.15.60:6801/3490482162","cluster_addr":"172.21.15.60:6803/3490482162","heartbeat_back_addr":"172.21.15.60:6807/3490482162","heartbeat_front_addr":"172.21.15.60:6805/3490482162","state":["exists","up"]},{"osd":7,"uuid":"544f8a91-ce82-4855-85eb-ea814eb3d058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6817","nonce":1315554966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6819","nonce":1315554966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6823","nonce":1315554966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6821","nonce":1315554966}]},"public_addr":"172.21.15.60:6817/1315554966","cluster_addr":"172.21.15.60:6819/1315554966","heartbeat_back_addr":"172.21.15.60:6823/1315554966","heartbeat_front_addr":"172.21.15.60:6821/1315554966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.514872+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.912129+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.745154+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:16.758871+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.287462+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:18.031636+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.055855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.524609+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[0,1]},{"pgid":"2.6","osds":[2,4]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:27:08.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T23:27:08.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T23:27:08.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T23:27:08.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T23:27:08.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-21T23:27:08.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-21T23:27:08.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-21T23:27:08.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-21T23:27:08.899 INFO:teuthology.orchestra.run.smithi031.stdout:64424509466 2024-09-21T23:27:08.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T23:27:08.909 INFO:teuthology.orchestra.run.smithi031.stdout:55834574873 2024-09-21T23:27:08.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T23:27:08.913 INFO:teuthology.orchestra.run.smithi031.stdout:60129542169 2024-09-21T23:27:08.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T23:27:08.914 INFO:teuthology.orchestra.run.smithi031.stdout:55834574873 2024-09-21T23:27:08.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-21T23:27:08.915 INFO:teuthology.orchestra.run.smithi031.stdout:64424509466 2024-09-21T23:27:08.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T23:27:08.917 INFO:teuthology.orchestra.run.smithi031.stdout:64424509467 2024-09-21T23:27:08.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T23:27:08.920 INFO:teuthology.orchestra.run.smithi031.stdout:64424509467 2024-09-21T23:27:08.921 INFO:teuthology.orchestra.run.smithi031.stdout:60129542170 2024-09-21T23:27:08.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T23:27:08.922 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T23:27:09.480 INFO:teuthology.orchestra.run.smithi031.stdout:64424509465 2024-09-21T23:27:09.500 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509466 got 64424509465 for osd.0 2024-09-21T23:27:09.505 INFO:teuthology.orchestra.run.smithi031.stdout:64424509465 2024-09-21T23:27:09.512 INFO:teuthology.orchestra.run.smithi031.stdout:60129542168 2024-09-21T23:27:09.517 INFO:teuthology.orchestra.run.smithi031.stdout:55834574872 2024-09-21T23:27:09.522 INFO:teuthology.orchestra.run.smithi031.stdout:55834574872 2024-09-21T23:27:09.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509466 got 64424509465 for osd.2 2024-09-21T23:27:09.536 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542169 got 60129542168 for osd.4 2024-09-21T23:27:09.537 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574873 got 55834574872 for osd.3 2024-09-21T23:27:09.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574873 got 55834574872 for osd.6 2024-09-21T23:27:09.544 INFO:teuthology.orchestra.run.smithi031.stdout:60129542169 2024-09-21T23:27:09.544 INFO:teuthology.orchestra.run.smithi031.stdout:64424509466 2024-09-21T23:27:09.549 INFO:teuthology.orchestra.run.smithi031.stdout:64424509466 2024-09-21T23:27:09.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542170 got 60129542169 for osd.7 2024-09-21T23:27:09.560 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509467 got 64424509466 for osd.5 2024-09-21T23:27:09.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509467 got 64424509466 for osd.1 2024-09-21T23:27:10.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T23:27:10.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T23:27:10.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T23:27:10.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T23:27:10.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-21T23:27:10.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T23:27:10.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T23:27:10.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T23:27:11.090 INFO:teuthology.orchestra.run.smithi031.stdout:64424509466 2024-09-21T23:27:11.111 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509466 got 64424509466 for osd.0 2024-09-21T23:27:11.111 DEBUG:teuthology.parallel:result is None 2024-09-21T23:27:11.125 INFO:teuthology.orchestra.run.smithi031.stdout:60129542169 2024-09-21T23:27:11.133 INFO:teuthology.orchestra.run.smithi031.stdout:64424509466 2024-09-21T23:27:11.137 INFO:teuthology.orchestra.run.smithi031.stdout:55834574874 2024-09-21T23:27:11.145 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542169 got 60129542169 for osd.4 2024-09-21T23:27:11.145 INFO:teuthology.orchestra.run.smithi031.stdout:55834574874 2024-09-21T23:27:11.146 DEBUG:teuthology.parallel:result is None 2024-09-21T23:27:11.153 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509466 got 64424509466 for osd.2 2024-09-21T23:27:11.154 DEBUG:teuthology.parallel:result is None 2024-09-21T23:27:11.157 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574873 got 55834574874 for osd.3 2024-09-21T23:27:11.157 DEBUG:teuthology.parallel:result is None 2024-09-21T23:27:11.165 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574873 got 55834574874 for osd.6 2024-09-21T23:27:11.166 DEBUG:teuthology.parallel:result is None 2024-09-21T23:27:11.171 INFO:teuthology.orchestra.run.smithi031.stdout:64424509467 2024-09-21T23:27:11.173 INFO:teuthology.orchestra.run.smithi031.stdout:60129542170 2024-09-21T23:27:11.177 INFO:teuthology.orchestra.run.smithi031.stdout:64424509467 2024-09-21T23:27:11.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509467 got 64424509467 for osd.5 2024-09-21T23:27:11.188 DEBUG:teuthology.parallel:result is None 2024-09-21T23:27:11.189 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542170 got 60129542170 for osd.7 2024-09-21T23:27:11.189 DEBUG:teuthology.parallel:result is None 2024-09-21T23:27:11.192 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509467 got 64424509467 for osd.1 2024-09-21T23:27:11.192 DEBUG:teuthology.parallel:result is None 2024-09-21T23:27:11.192 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T23:27:11.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:27:11.522 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:27:11.523 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:27:11.534 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-21T23:27:09.793865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2964,"num_read_kb":2805,"num_write":579,"num_write_kb":2844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1095,"ondisk_log_size":1095,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256496,"kb_used_data":6952,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718224,"statfs":{"total":773094113280,"available":772831461376,"internally_reserved":0,"allocated":7118848,"data_stored":4162358,"data_compressed":19550,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.789972"},"pg_stats":[{"pgid":"2.7","version":"59'114","reported_seq":419,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-21T23:27:07.730082+0000","last_change":"2024-09-21T23:26:48.002220+0000","last_active":"2024-09-21T23:27:07.730082+0000","last_peered":"2024-09-21T23:27:07.730082+0000","last_clean":"2024-09-21T23:27:07.730082+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:27:07.730082+0000","last_undegraded":"2024-09-21T23:27:07.730082+0000","last_fullsized":"2024-09-21T23:27:07.730082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:25:03.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002039819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":357,"num_read_kb":326,"num_write":43,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"59'86","reported_seq":273,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027088+0000","last_change":"2024-09-21T23:26:47.912302+0000","last_active":"2024-09-21T23:27:08.027088+0000","last_peered":"2024-09-21T23:27:08.027088+0000","last_clean":"2024-09-21T23:27:08.027088+0000","last_became_active":"2024-09-21T23:26:17.546387+0000","last_became_peered":"2024-09-21T23:26:17.546387+0000","last_unstale":"2024-09-21T23:27:08.027088+0000","last_undegraded":"2024-09-21T23:27:08.027088+0000","last_fullsized":"2024-09-21T23:27:08.027088+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:48:31.573163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000423046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":218,"num_read_kb":168,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"59'155","reported_seq":510,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424792+0000","last_change":"2024-09-21T23:26:48.002349+0000","last_active":"2024-09-21T23:27:08.424792+0000","last_peered":"2024-09-21T23:27:08.424792+0000","last_clean":"2024-09-21T23:27:08.424792+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:27:08.424792+0000","last_undegraded":"2024-09-21T23:27:08.424792+0000","last_fullsized":"2024-09-21T23:27:08.424792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:16:31.020421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018417240000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":432,"num_read_kb":441,"num_write":88,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"59'146","reported_seq":541,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027077+0000","last_change":"2024-09-21T23:26:47.912580+0000","last_active":"2024-09-21T23:27:08.027077+0000","last_peered":"2024-09-21T23:27:08.027077+0000","last_clean":"2024-09-21T23:27:08.027077+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:27:08.027077+0000","last_undegraded":"2024-09-21T23:27:08.027077+0000","last_fullsized":"2024-09-21T23:27:08.027077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:56:58.087164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055139900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":636,"num_read_kb":592,"num_write":64,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'154","reported_seq":657,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197175+0000","last_change":"2024-09-21T23:26:47.963590+0000","last_active":"2024-09-21T23:27:08.197175+0000","last_peered":"2024-09-21T23:27:08.197175+0000","last_clean":"2024-09-21T23:27:08.197175+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:27:08.197175+0000","last_undegraded":"2024-09-21T23:27:08.197175+0000","last_fullsized":"2024-09-21T23:27:08.197175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:24:23.124579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063174900000000003,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":655,"num_read_kb":658,"num_write":67,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"59'124","reported_seq":462,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027056+0000","last_change":"2024-09-21T23:26:47.912343+0000","last_active":"2024-09-21T23:27:08.027056+0000","last_peered":"2024-09-21T23:27:08.027056+0000","last_clean":"2024-09-21T23:27:08.027056+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:27:08.027056+0000","last_undegraded":"2024-09-21T23:27:08.027056+0000","last_fullsized":"2024-09-21T23:27:08.027056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:24:00.919645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046220700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":385,"num_read_kb":351,"num_write":58,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"59'80","reported_seq":242,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424773+0000","last_change":"2024-09-21T23:26:48.000677+0000","last_active":"2024-09-21T23:27:08.424773+0000","last_peered":"2024-09-21T23:27:08.424773+0000","last_clean":"2024-09-21T23:27:08.424773+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:27:08.424773+0000","last_undegraded":"2024-09-21T23:27:08.424773+0000","last_fullsized":"2024-09-21T23:27:08.424773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:57:46.982872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000365359,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":22,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"59'119","reported_seq":324,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197178+0000","last_change":"2024-09-21T23:26:47.963368+0000","last_active":"2024-09-21T23:27:08.197178+0000","last_peered":"2024-09-21T23:27:08.197178+0000","last_clean":"2024-09-21T23:27:08.197178+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:27:08.197178+0000","last_undegraded":"2024-09-21T23:27:08.197178+0000","last_fullsized":"2024-09-21T23:27:08.197178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:15:22.026082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036157399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":132,"num_write":67,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":227,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.450206+0000","last_change":"2024-09-21T23:26:54.862240+0000","last_active":"2024-09-21T23:27:08.450206+0000","last_peered":"2024-09-21T23:27:08.450206+0000","last_clean":"2024-09-21T23:27:08.450206+0000","last_became_active":"2024-09-21T23:26:54.861854+0000","last_became_peered":"2024-09-21T23:26:54.861854+0000","last_unstale":"2024-09-21T23:27:08.450206+0000","last_undegraded":"2024-09-21T23:27:08.450206+0000","last_fullsized":"2024-09-21T23:27:08.450206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:03:03.687216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2848,"num_read_kb":2705,"num_write":420,"num_write_kb":428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":978,"ondisk_log_size":978,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":47842,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542170,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32048,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":872448,"data_stored":369944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509467,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":888832,"data_stored":375471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542169,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509466,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509467,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":929792,"data_stored":965839,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509466,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":913408,"data_stored":960312,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":872448,"data_stored":369944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":872448,"data_stored":369944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:27:11.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:27:11.822 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:27:11.822 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:27:11.835 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-21T23:27:11.794248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2964,"num_read_kb":2805,"num_write":579,"num_write_kb":2844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1095,"ondisk_log_size":1095,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256504,"kb_used_data":6976,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718216,"statfs":{"total":773094113280,"available":772831453184,"internally_reserved":0,"allocated":7143424,"data_stored":4173374,"data_compressed":19550,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001017"},"pg_stats":[{"pgid":"2.7","version":"59'114","reported_seq":419,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-21T23:27:07.730082+0000","last_change":"2024-09-21T23:26:48.002220+0000","last_active":"2024-09-21T23:27:07.730082+0000","last_peered":"2024-09-21T23:27:07.730082+0000","last_clean":"2024-09-21T23:27:07.730082+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:27:07.730082+0000","last_undegraded":"2024-09-21T23:27:07.730082+0000","last_fullsized":"2024-09-21T23:27:07.730082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:25:03.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002039819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":357,"num_read_kb":326,"num_write":43,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"59'86","reported_seq":273,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027088+0000","last_change":"2024-09-21T23:26:47.912302+0000","last_active":"2024-09-21T23:27:08.027088+0000","last_peered":"2024-09-21T23:27:08.027088+0000","last_clean":"2024-09-21T23:27:08.027088+0000","last_became_active":"2024-09-21T23:26:17.546387+0000","last_became_peered":"2024-09-21T23:26:17.546387+0000","last_unstale":"2024-09-21T23:27:08.027088+0000","last_undegraded":"2024-09-21T23:27:08.027088+0000","last_fullsized":"2024-09-21T23:27:08.027088+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:48:31.573163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000423046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":218,"num_read_kb":168,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"59'155","reported_seq":510,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424792+0000","last_change":"2024-09-21T23:26:48.002349+0000","last_active":"2024-09-21T23:27:08.424792+0000","last_peered":"2024-09-21T23:27:08.424792+0000","last_clean":"2024-09-21T23:27:08.424792+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:27:08.424792+0000","last_undegraded":"2024-09-21T23:27:08.424792+0000","last_fullsized":"2024-09-21T23:27:08.424792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:16:31.020421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018417240000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":432,"num_read_kb":441,"num_write":88,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"59'146","reported_seq":541,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027077+0000","last_change":"2024-09-21T23:26:47.912580+0000","last_active":"2024-09-21T23:27:08.027077+0000","last_peered":"2024-09-21T23:27:08.027077+0000","last_clean":"2024-09-21T23:27:08.027077+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:27:08.027077+0000","last_undegraded":"2024-09-21T23:27:08.027077+0000","last_fullsized":"2024-09-21T23:27:08.027077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:56:58.087164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055139900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":636,"num_read_kb":592,"num_write":64,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'154","reported_seq":657,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197175+0000","last_change":"2024-09-21T23:26:47.963590+0000","last_active":"2024-09-21T23:27:08.197175+0000","last_peered":"2024-09-21T23:27:08.197175+0000","last_clean":"2024-09-21T23:27:08.197175+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:27:08.197175+0000","last_undegraded":"2024-09-21T23:27:08.197175+0000","last_fullsized":"2024-09-21T23:27:08.197175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:24:23.124579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063174900000000003,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":655,"num_read_kb":658,"num_write":67,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"59'124","reported_seq":462,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027056+0000","last_change":"2024-09-21T23:26:47.912343+0000","last_active":"2024-09-21T23:27:08.027056+0000","last_peered":"2024-09-21T23:27:08.027056+0000","last_clean":"2024-09-21T23:27:08.027056+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:27:08.027056+0000","last_undegraded":"2024-09-21T23:27:08.027056+0000","last_fullsized":"2024-09-21T23:27:08.027056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:24:00.919645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046220700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":385,"num_read_kb":351,"num_write":58,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"59'80","reported_seq":242,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424773+0000","last_change":"2024-09-21T23:26:48.000677+0000","last_active":"2024-09-21T23:27:08.424773+0000","last_peered":"2024-09-21T23:27:08.424773+0000","last_clean":"2024-09-21T23:27:08.424773+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:27:08.424773+0000","last_undegraded":"2024-09-21T23:27:08.424773+0000","last_fullsized":"2024-09-21T23:27:08.424773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:57:46.982872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000365359,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":22,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"59'119","reported_seq":324,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197178+0000","last_change":"2024-09-21T23:26:47.963368+0000","last_active":"2024-09-21T23:27:08.197178+0000","last_peered":"2024-09-21T23:27:08.197178+0000","last_clean":"2024-09-21T23:27:08.197178+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:27:08.197178+0000","last_undegraded":"2024-09-21T23:27:08.197178+0000","last_fullsized":"2024-09-21T23:27:08.197178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:15:22.026082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036157399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":132,"num_write":67,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":227,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.450206+0000","last_change":"2024-09-21T23:26:54.862240+0000","last_active":"2024-09-21T23:27:08.450206+0000","last_peered":"2024-09-21T23:27:08.450206+0000","last_clean":"2024-09-21T23:27:08.450206+0000","last_became_active":"2024-09-21T23:26:54.861854+0000","last_became_peered":"2024-09-21T23:26:54.861854+0000","last_unstale":"2024-09-21T23:27:08.450206+0000","last_undegraded":"2024-09-21T23:27:08.450206+0000","last_fullsized":"2024-09-21T23:27:08.450206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:03:03.687216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2848,"num_read_kb":2705,"num_write":420,"num_write_kb":428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":978,"ondisk_log_size":978,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":47842,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542171,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32052,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339788,"statfs":{"total":96636764160,"available":96603942912,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509468,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":888832,"data_stored":375471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509468,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":929792,"data_stored":965839,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":925696,"data_stored":965820,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":872448,"data_stored":369944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":872448,"data_stored":369944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:27:11.836 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-21T23:27:11.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-21T23:27:12.148 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-21T23:27:12.158 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T23:27:12.158 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:27:12.445 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:27:12.446 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:27:12.457 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-21T23:27:11.794248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2964,"num_read_kb":2805,"num_write":579,"num_write_kb":2844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1095,"ondisk_log_size":1095,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256504,"kb_used_data":6976,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718216,"statfs":{"total":773094113280,"available":772831453184,"internally_reserved":0,"allocated":7143424,"data_stored":4173374,"data_compressed":19550,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001017"},"pg_stats":[{"pgid":"2.7","version":"59'114","reported_seq":419,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-21T23:27:07.730082+0000","last_change":"2024-09-21T23:26:48.002220+0000","last_active":"2024-09-21T23:27:07.730082+0000","last_peered":"2024-09-21T23:27:07.730082+0000","last_clean":"2024-09-21T23:27:07.730082+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:27:07.730082+0000","last_undegraded":"2024-09-21T23:27:07.730082+0000","last_fullsized":"2024-09-21T23:27:07.730082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:25:03.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002039819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":357,"num_read_kb":326,"num_write":43,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"59'86","reported_seq":273,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027088+0000","last_change":"2024-09-21T23:26:47.912302+0000","last_active":"2024-09-21T23:27:08.027088+0000","last_peered":"2024-09-21T23:27:08.027088+0000","last_clean":"2024-09-21T23:27:08.027088+0000","last_became_active":"2024-09-21T23:26:17.546387+0000","last_became_peered":"2024-09-21T23:26:17.546387+0000","last_unstale":"2024-09-21T23:27:08.027088+0000","last_undegraded":"2024-09-21T23:27:08.027088+0000","last_fullsized":"2024-09-21T23:27:08.027088+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:48:31.573163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000423046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":218,"num_read_kb":168,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"59'155","reported_seq":510,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424792+0000","last_change":"2024-09-21T23:26:48.002349+0000","last_active":"2024-09-21T23:27:08.424792+0000","last_peered":"2024-09-21T23:27:08.424792+0000","last_clean":"2024-09-21T23:27:08.424792+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:27:08.424792+0000","last_undegraded":"2024-09-21T23:27:08.424792+0000","last_fullsized":"2024-09-21T23:27:08.424792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:16:31.020421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018417240000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":432,"num_read_kb":441,"num_write":88,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"59'146","reported_seq":541,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027077+0000","last_change":"2024-09-21T23:26:47.912580+0000","last_active":"2024-09-21T23:27:08.027077+0000","last_peered":"2024-09-21T23:27:08.027077+0000","last_clean":"2024-09-21T23:27:08.027077+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:27:08.027077+0000","last_undegraded":"2024-09-21T23:27:08.027077+0000","last_fullsized":"2024-09-21T23:27:08.027077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:56:58.087164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055139900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":636,"num_read_kb":592,"num_write":64,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'154","reported_seq":657,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197175+0000","last_change":"2024-09-21T23:26:47.963590+0000","last_active":"2024-09-21T23:27:08.197175+0000","last_peered":"2024-09-21T23:27:08.197175+0000","last_clean":"2024-09-21T23:27:08.197175+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:27:08.197175+0000","last_undegraded":"2024-09-21T23:27:08.197175+0000","last_fullsized":"2024-09-21T23:27:08.197175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:24:23.124579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063174900000000003,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":655,"num_read_kb":658,"num_write":67,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"59'124","reported_seq":462,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027056+0000","last_change":"2024-09-21T23:26:47.912343+0000","last_active":"2024-09-21T23:27:08.027056+0000","last_peered":"2024-09-21T23:27:08.027056+0000","last_clean":"2024-09-21T23:27:08.027056+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:27:08.027056+0000","last_undegraded":"2024-09-21T23:27:08.027056+0000","last_fullsized":"2024-09-21T23:27:08.027056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:24:00.919645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046220700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":385,"num_read_kb":351,"num_write":58,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"59'80","reported_seq":242,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424773+0000","last_change":"2024-09-21T23:26:48.000677+0000","last_active":"2024-09-21T23:27:08.424773+0000","last_peered":"2024-09-21T23:27:08.424773+0000","last_clean":"2024-09-21T23:27:08.424773+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:27:08.424773+0000","last_undegraded":"2024-09-21T23:27:08.424773+0000","last_fullsized":"2024-09-21T23:27:08.424773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:57:46.982872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000365359,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":22,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"59'119","reported_seq":324,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197178+0000","last_change":"2024-09-21T23:26:47.963368+0000","last_active":"2024-09-21T23:27:08.197178+0000","last_peered":"2024-09-21T23:27:08.197178+0000","last_clean":"2024-09-21T23:27:08.197178+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:27:08.197178+0000","last_undegraded":"2024-09-21T23:27:08.197178+0000","last_fullsized":"2024-09-21T23:27:08.197178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:15:22.026082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036157399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":132,"num_write":67,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":227,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.450206+0000","last_change":"2024-09-21T23:26:54.862240+0000","last_active":"2024-09-21T23:27:08.450206+0000","last_peered":"2024-09-21T23:27:08.450206+0000","last_clean":"2024-09-21T23:27:08.450206+0000","last_became_active":"2024-09-21T23:26:54.861854+0000","last_became_peered":"2024-09-21T23:26:54.861854+0000","last_unstale":"2024-09-21T23:27:08.450206+0000","last_undegraded":"2024-09-21T23:27:08.450206+0000","last_fullsized":"2024-09-21T23:27:08.450206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:03:03.687216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2848,"num_read_kb":2705,"num_write":420,"num_write_kb":428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":978,"ondisk_log_size":978,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":47842,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542171,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32052,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339788,"statfs":{"total":96636764160,"available":96603942912,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509468,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":888832,"data_stored":375471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509468,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":929792,"data_stored":965839,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":925696,"data_stored":965820,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":872448,"data_stored":369944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":872448,"data_stored":369944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:27:12.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:27:12.744 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:27:12.745 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:27:12.756 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-21T23:27:11.794248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2964,"num_read_kb":2805,"num_write":579,"num_write_kb":2844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1095,"ondisk_log_size":1095,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256504,"kb_used_data":6976,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718216,"statfs":{"total":773094113280,"available":772831453184,"internally_reserved":0,"allocated":7143424,"data_stored":4173374,"data_compressed":19550,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001017"},"pg_stats":[{"pgid":"2.7","version":"59'114","reported_seq":419,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-21T23:27:07.730082+0000","last_change":"2024-09-21T23:26:48.002220+0000","last_active":"2024-09-21T23:27:07.730082+0000","last_peered":"2024-09-21T23:27:07.730082+0000","last_clean":"2024-09-21T23:27:07.730082+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:27:07.730082+0000","last_undegraded":"2024-09-21T23:27:07.730082+0000","last_fullsized":"2024-09-21T23:27:07.730082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:25:03.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002039819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":357,"num_read_kb":326,"num_write":43,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"59'86","reported_seq":273,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027088+0000","last_change":"2024-09-21T23:26:47.912302+0000","last_active":"2024-09-21T23:27:08.027088+0000","last_peered":"2024-09-21T23:27:08.027088+0000","last_clean":"2024-09-21T23:27:08.027088+0000","last_became_active":"2024-09-21T23:26:17.546387+0000","last_became_peered":"2024-09-21T23:26:17.546387+0000","last_unstale":"2024-09-21T23:27:08.027088+0000","last_undegraded":"2024-09-21T23:27:08.027088+0000","last_fullsized":"2024-09-21T23:27:08.027088+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:48:31.573163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000423046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":218,"num_read_kb":168,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"59'155","reported_seq":510,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424792+0000","last_change":"2024-09-21T23:26:48.002349+0000","last_active":"2024-09-21T23:27:08.424792+0000","last_peered":"2024-09-21T23:27:08.424792+0000","last_clean":"2024-09-21T23:27:08.424792+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:27:08.424792+0000","last_undegraded":"2024-09-21T23:27:08.424792+0000","last_fullsized":"2024-09-21T23:27:08.424792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:16:31.020421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018417240000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":432,"num_read_kb":441,"num_write":88,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"59'146","reported_seq":541,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027077+0000","last_change":"2024-09-21T23:26:47.912580+0000","last_active":"2024-09-21T23:27:08.027077+0000","last_peered":"2024-09-21T23:27:08.027077+0000","last_clean":"2024-09-21T23:27:08.027077+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:27:08.027077+0000","last_undegraded":"2024-09-21T23:27:08.027077+0000","last_fullsized":"2024-09-21T23:27:08.027077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:56:58.087164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055139900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":636,"num_read_kb":592,"num_write":64,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'154","reported_seq":657,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197175+0000","last_change":"2024-09-21T23:26:47.963590+0000","last_active":"2024-09-21T23:27:08.197175+0000","last_peered":"2024-09-21T23:27:08.197175+0000","last_clean":"2024-09-21T23:27:08.197175+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:27:08.197175+0000","last_undegraded":"2024-09-21T23:27:08.197175+0000","last_fullsized":"2024-09-21T23:27:08.197175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:24:23.124579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063174900000000003,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":655,"num_read_kb":658,"num_write":67,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"59'124","reported_seq":462,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027056+0000","last_change":"2024-09-21T23:26:47.912343+0000","last_active":"2024-09-21T23:27:08.027056+0000","last_peered":"2024-09-21T23:27:08.027056+0000","last_clean":"2024-09-21T23:27:08.027056+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:27:08.027056+0000","last_undegraded":"2024-09-21T23:27:08.027056+0000","last_fullsized":"2024-09-21T23:27:08.027056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:24:00.919645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046220700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":385,"num_read_kb":351,"num_write":58,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"59'80","reported_seq":242,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424773+0000","last_change":"2024-09-21T23:26:48.000677+0000","last_active":"2024-09-21T23:27:08.424773+0000","last_peered":"2024-09-21T23:27:08.424773+0000","last_clean":"2024-09-21T23:27:08.424773+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:27:08.424773+0000","last_undegraded":"2024-09-21T23:27:08.424773+0000","last_fullsized":"2024-09-21T23:27:08.424773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:57:46.982872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000365359,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":22,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"59'119","reported_seq":324,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197178+0000","last_change":"2024-09-21T23:26:47.963368+0000","last_active":"2024-09-21T23:27:08.197178+0000","last_peered":"2024-09-21T23:27:08.197178+0000","last_clean":"2024-09-21T23:27:08.197178+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:27:08.197178+0000","last_undegraded":"2024-09-21T23:27:08.197178+0000","last_fullsized":"2024-09-21T23:27:08.197178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:15:22.026082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036157399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":132,"num_write":67,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":227,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.450206+0000","last_change":"2024-09-21T23:26:54.862240+0000","last_active":"2024-09-21T23:27:08.450206+0000","last_peered":"2024-09-21T23:27:08.450206+0000","last_clean":"2024-09-21T23:27:08.450206+0000","last_became_active":"2024-09-21T23:26:54.861854+0000","last_became_peered":"2024-09-21T23:26:54.861854+0000","last_unstale":"2024-09-21T23:27:08.450206+0000","last_undegraded":"2024-09-21T23:27:08.450206+0000","last_fullsized":"2024-09-21T23:27:08.450206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:03:03.687216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2848,"num_read_kb":2705,"num_write":420,"num_write_kb":428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":978,"ondisk_log_size":978,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":47842,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542171,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32052,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339788,"statfs":{"total":96636764160,"available":96603942912,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509468,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":888832,"data_stored":375471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509468,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":929792,"data_stored":965839,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":925696,"data_stored":965820,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":872448,"data_stored":369944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":872448,"data_stored":369944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:27:12.757 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T23:27:12.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:27:13.046 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:27:13.047 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:27:13.060 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-21T23:27:11.794248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2964,"num_read_kb":2805,"num_write":579,"num_write_kb":2844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1095,"ondisk_log_size":1095,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256504,"kb_used_data":6976,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718216,"statfs":{"total":773094113280,"available":772831453184,"internally_reserved":0,"allocated":7143424,"data_stored":4173374,"data_compressed":19550,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001017"},"pg_stats":[{"pgid":"2.7","version":"59'114","reported_seq":419,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-21T23:27:07.730082+0000","last_change":"2024-09-21T23:26:48.002220+0000","last_active":"2024-09-21T23:27:07.730082+0000","last_peered":"2024-09-21T23:27:07.730082+0000","last_clean":"2024-09-21T23:27:07.730082+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:27:07.730082+0000","last_undegraded":"2024-09-21T23:27:07.730082+0000","last_fullsized":"2024-09-21T23:27:07.730082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:25:03.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002039819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":357,"num_read_kb":326,"num_write":43,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"59'86","reported_seq":273,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027088+0000","last_change":"2024-09-21T23:26:47.912302+0000","last_active":"2024-09-21T23:27:08.027088+0000","last_peered":"2024-09-21T23:27:08.027088+0000","last_clean":"2024-09-21T23:27:08.027088+0000","last_became_active":"2024-09-21T23:26:17.546387+0000","last_became_peered":"2024-09-21T23:26:17.546387+0000","last_unstale":"2024-09-21T23:27:08.027088+0000","last_undegraded":"2024-09-21T23:27:08.027088+0000","last_fullsized":"2024-09-21T23:27:08.027088+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:48:31.573163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000423046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":218,"num_read_kb":168,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"59'155","reported_seq":510,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424792+0000","last_change":"2024-09-21T23:26:48.002349+0000","last_active":"2024-09-21T23:27:08.424792+0000","last_peered":"2024-09-21T23:27:08.424792+0000","last_clean":"2024-09-21T23:27:08.424792+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:27:08.424792+0000","last_undegraded":"2024-09-21T23:27:08.424792+0000","last_fullsized":"2024-09-21T23:27:08.424792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:16:31.020421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018417240000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":432,"num_read_kb":441,"num_write":88,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"59'146","reported_seq":541,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027077+0000","last_change":"2024-09-21T23:26:47.912580+0000","last_active":"2024-09-21T23:27:08.027077+0000","last_peered":"2024-09-21T23:27:08.027077+0000","last_clean":"2024-09-21T23:27:08.027077+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:27:08.027077+0000","last_undegraded":"2024-09-21T23:27:08.027077+0000","last_fullsized":"2024-09-21T23:27:08.027077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:56:58.087164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055139900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":636,"num_read_kb":592,"num_write":64,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'154","reported_seq":657,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197175+0000","last_change":"2024-09-21T23:26:47.963590+0000","last_active":"2024-09-21T23:27:08.197175+0000","last_peered":"2024-09-21T23:27:08.197175+0000","last_clean":"2024-09-21T23:27:08.197175+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:27:08.197175+0000","last_undegraded":"2024-09-21T23:27:08.197175+0000","last_fullsized":"2024-09-21T23:27:08.197175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:24:23.124579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063174900000000003,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":655,"num_read_kb":658,"num_write":67,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"59'124","reported_seq":462,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027056+0000","last_change":"2024-09-21T23:26:47.912343+0000","last_active":"2024-09-21T23:27:08.027056+0000","last_peered":"2024-09-21T23:27:08.027056+0000","last_clean":"2024-09-21T23:27:08.027056+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:27:08.027056+0000","last_undegraded":"2024-09-21T23:27:08.027056+0000","last_fullsized":"2024-09-21T23:27:08.027056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:24:00.919645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046220700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":385,"num_read_kb":351,"num_write":58,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"59'80","reported_seq":242,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424773+0000","last_change":"2024-09-21T23:26:48.000677+0000","last_active":"2024-09-21T23:27:08.424773+0000","last_peered":"2024-09-21T23:27:08.424773+0000","last_clean":"2024-09-21T23:27:08.424773+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:27:08.424773+0000","last_undegraded":"2024-09-21T23:27:08.424773+0000","last_fullsized":"2024-09-21T23:27:08.424773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:57:46.982872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000365359,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":22,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"59'119","reported_seq":324,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197178+0000","last_change":"2024-09-21T23:26:47.963368+0000","last_active":"2024-09-21T23:27:08.197178+0000","last_peered":"2024-09-21T23:27:08.197178+0000","last_clean":"2024-09-21T23:27:08.197178+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:27:08.197178+0000","last_undegraded":"2024-09-21T23:27:08.197178+0000","last_fullsized":"2024-09-21T23:27:08.197178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:15:22.026082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036157399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":132,"num_write":67,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":227,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.450206+0000","last_change":"2024-09-21T23:26:54.862240+0000","last_active":"2024-09-21T23:27:08.450206+0000","last_peered":"2024-09-21T23:27:08.450206+0000","last_clean":"2024-09-21T23:27:08.450206+0000","last_became_active":"2024-09-21T23:26:54.861854+0000","last_became_peered":"2024-09-21T23:26:54.861854+0000","last_unstale":"2024-09-21T23:27:08.450206+0000","last_undegraded":"2024-09-21T23:27:08.450206+0000","last_fullsized":"2024-09-21T23:27:08.450206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:20.126996+0000","last_clean_scrub_stamp":"2024-09-21T23:25:20.126996+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:03:03.687216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2848,"num_read_kb":2705,"num_write":420,"num_write_kb":428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":978,"ondisk_log_size":978,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":47842,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542171,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32052,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339788,"statfs":{"total":96636764160,"available":96603942912,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509468,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":888832,"data_stored":375471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509468,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":929792,"data_stored":965839,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":925696,"data_stored":965820,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32040,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339800,"statfs":{"total":96636764160,"available":96603955200,"internally_reserved":0,"allocated":872448,"data_stored":369944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":872448,"data_stored":369944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:27:13.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T23:27:13.354 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:27:13.354 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":76,"fsid":"d51cc561-ddb8-41ef-b61c-7c6d698e4dd0","created":"2024-09-21T23:25:09.056809+0000","modified":"2024-09-21T23:27:08.020330+0000","last_up_change":"2024-09-21T23:25:20.126996+0000","last_in_change":"2024-09-21T23:25:11.226810+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:25:19.818212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T23:25:26.968096+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e46cdf8b-8ba4-45a0-ab3c-dcf78d45c7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6817","nonce":2339966024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6819","nonce":2339966024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6823","nonce":2339966024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2339966024},{"type":"v1","addr":"172.21.15.31:6821","nonce":2339966024}]},"public_addr":"172.21.15.31:6817/2339966024","cluster_addr":"172.21.15.31:6819/2339966024","heartbeat_back_addr":"172.21.15.31:6823/2339966024","heartbeat_front_addr":"172.21.15.31:6821/2339966024","state":["exists","up"]},{"osd":1,"uuid":"88ee06a3-9a6a-41e8-9327-a7a61304c03d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6825","nonce":2889649822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6827","nonce":2889649822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6831","nonce":2889649822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":2889649822},{"type":"v1","addr":"172.21.15.31:6829","nonce":2889649822}]},"public_addr":"172.21.15.31:6825/2889649822","cluster_addr":"172.21.15.31:6827/2889649822","heartbeat_back_addr":"172.21.15.31:6831/2889649822","heartbeat_front_addr":"172.21.15.31:6829/2889649822","state":["exists","up"]},{"osd":2,"uuid":"f4e3b1de-d3dc-424e-8bdc-a9fd0207fd05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6809","nonce":864917651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6811","nonce":864917651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6815","nonce":864917651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":864917651},{"type":"v1","addr":"172.21.15.31:6813","nonce":864917651}]},"public_addr":"172.21.15.31:6809/864917651","cluster_addr":"172.21.15.31:6811/864917651","heartbeat_back_addr":"172.21.15.31:6815/864917651","heartbeat_front_addr":"172.21.15.31:6813/864917651","state":["exists","up"]},{"osd":3,"uuid":"562e8c30-0622-4cb3-b5d1-f15b42a444bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6801","nonce":1713632981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6803","nonce":1713632981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6807","nonce":1713632981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1713632981},{"type":"v1","addr":"172.21.15.31:6805","nonce":1713632981}]},"public_addr":"172.21.15.31:6801/1713632981","cluster_addr":"172.21.15.31:6803/1713632981","heartbeat_back_addr":"172.21.15.31:6807/1713632981","heartbeat_front_addr":"172.21.15.31:6805/1713632981","state":["exists","up"]},{"osd":4,"uuid":"7030ff68-4556-4ad5-abc9-be758f1caa85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6809","nonce":2660753894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6811","nonce":2660753894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6815","nonce":2660753894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":2660753894},{"type":"v1","addr":"172.21.15.60:6813","nonce":2660753894}]},"public_addr":"172.21.15.60:6809/2660753894","cluster_addr":"172.21.15.60:6811/2660753894","heartbeat_back_addr":"172.21.15.60:6815/2660753894","heartbeat_front_addr":"172.21.15.60:6813/2660753894","state":["exists","up"]},{"osd":5,"uuid":"7169a163-f067-4385-b734-f58a3ba9e89e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6825","nonce":2320687649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6827","nonce":2320687649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6831","nonce":2320687649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":2320687649},{"type":"v1","addr":"172.21.15.60:6829","nonce":2320687649}]},"public_addr":"172.21.15.60:6825/2320687649","cluster_addr":"172.21.15.60:6827/2320687649","heartbeat_back_addr":"172.21.15.60:6831/2320687649","heartbeat_front_addr":"172.21.15.60:6829/2320687649","state":["exists","up"]},{"osd":6,"uuid":"e511295d-2177-4dcb-a74b-b16b92534df6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6801","nonce":3490482162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6803","nonce":3490482162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6807","nonce":3490482162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3490482162},{"type":"v1","addr":"172.21.15.60:6805","nonce":3490482162}]},"public_addr":"172.21.15.60:6801/3490482162","cluster_addr":"172.21.15.60:6803/3490482162","heartbeat_back_addr":"172.21.15.60:6807/3490482162","heartbeat_front_addr":"172.21.15.60:6805/3490482162","state":["exists","up"]},{"osd":7,"uuid":"544f8a91-ce82-4855-85eb-ea814eb3d058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6817","nonce":1315554966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6819","nonce":1315554966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6823","nonce":1315554966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":1315554966},{"type":"v1","addr":"172.21.15.60:6821","nonce":1315554966}]},"public_addr":"172.21.15.60:6817/1315554966","cluster_addr":"172.21.15.60:6819/1315554966","heartbeat_back_addr":"172.21.15.60:6823/1315554966","heartbeat_front_addr":"172.21.15.60:6821/1315554966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.514872+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.912129+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.745154+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:16.758871+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.287462+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:18.031636+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.055855+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:25:17.524609+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[0,1]},{"pgid":"2.6","osds":[2,4]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:27:14.368 INFO:tasks.ceph:Scrubbing osd.0 2024-09-21T23:27:14.368 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-21T23:27:14.508 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:27:14.508 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T23:27:14.509 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:27:14.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-21T23:27:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 0 to deep-scrub 2024-09-21T23:27:14.825 INFO:tasks.ceph:Scrubbing osd.1 2024-09-21T23:27:14.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-21T23:27:14.953 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:27:14.953 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T23:27:14.954 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:27:14.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-21T23:27:15.254 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 1 to deep-scrub 2024-09-21T23:27:15.267 INFO:tasks.ceph:Scrubbing osd.2 2024-09-21T23:27:15.268 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-21T23:27:15.394 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:27:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T23:27:15.394 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:27:15.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-21T23:27:15.699 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 2 to deep-scrub 2024-09-21T23:27:15.712 INFO:tasks.ceph:Scrubbing osd.3 2024-09-21T23:27:15.712 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-21T23:27:15.838 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:27:15.838 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T23:27:15.838 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:27:15.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-21T23:27:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 3 to deep-scrub 2024-09-21T23:27:16.159 INFO:tasks.ceph:Scrubbing osd.4 2024-09-21T23:27:16.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-21T23:27:16.287 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:27:16.287 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T23:27:16.287 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:27:16.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-21T23:27:16.588 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 4 to deep-scrub 2024-09-21T23:27:16.601 INFO:tasks.ceph:Scrubbing osd.5 2024-09-21T23:27:16.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-21T23:27:16.729 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:27:16.729 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T23:27:16.729 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:27:16.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-21T23:27:17.029 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 5 to deep-scrub 2024-09-21T23:27:17.042 INFO:tasks.ceph:Scrubbing osd.6 2024-09-21T23:27:17.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-21T23:27:17.171 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:27:17.171 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T23:27:17.171 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:27:17.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-21T23:27:17.470 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 6 to deep-scrub 2024-09-21T23:27:17.484 INFO:tasks.ceph:Scrubbing osd.7 2024-09-21T23:27:17.484 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-21T23:27:17.611 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-21T23:27:17.611 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T23:27:17.611 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-21T23:27:17.621 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-21T23:27:17.913 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 7 to deep-scrub 2024-09-21T23:27:17.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:27:18.215 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:27:18.215 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:27:18.227 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-21T23:27:17.795234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2964,"num_read_kb":2805,"num_write":579,"num_write_kb":2844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1097,"ondisk_log_size":1097,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256512,"kb_used_data":7000,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718208,"statfs":{"total":773094113280,"available":772831444992,"internally_reserved":0,"allocated":7168000,"data_stored":4184390,"data_compressed":19550,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001672"},"pg_stats":[{"pgid":"2.7","version":"59'114","reported_seq":421,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:09.774356+0000","last_change":"2024-09-21T23:26:48.002220+0000","last_active":"2024-09-21T23:27:09.774356+0000","last_peered":"2024-09-21T23:27:09.774356+0000","last_clean":"2024-09-21T23:27:09.774356+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:27:09.774356+0000","last_undegraded":"2024-09-21T23:27:09.774356+0000","last_fullsized":"2024-09-21T23:27:09.774356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:25:03.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002039819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":357,"num_read_kb":326,"num_write":43,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"59'86","reported_seq":274,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027149+0000","last_change":"2024-09-21T23:26:47.912302+0000","last_active":"2024-09-21T23:27:08.027149+0000","last_peered":"2024-09-21T23:27:08.027149+0000","last_clean":"2024-09-21T23:27:08.027149+0000","last_became_active":"2024-09-21T23:26:17.546387+0000","last_became_peered":"2024-09-21T23:26:17.546387+0000","last_unstale":"2024-09-21T23:27:08.027149+0000","last_undegraded":"2024-09-21T23:27:08.027149+0000","last_fullsized":"2024-09-21T23:27:08.027149+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000423046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":218,"num_read_kb":168,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"59'155","reported_seq":510,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424792+0000","last_change":"2024-09-21T23:26:48.002349+0000","last_active":"2024-09-21T23:27:08.424792+0000","last_peered":"2024-09-21T23:27:08.424792+0000","last_clean":"2024-09-21T23:27:08.424792+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:27:08.424792+0000","last_undegraded":"2024-09-21T23:27:08.424792+0000","last_fullsized":"2024-09-21T23:27:08.424792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:16:31.020421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018417240000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":432,"num_read_kb":441,"num_write":88,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"59'146","reported_seq":550,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:15.272688+0000","last_change":"2024-09-21T23:27:15.272711+0000","last_active":"2024-09-21T23:27:15.272688+0000","last_peered":"2024-09-21T23:27:15.272688+0000","last_clean":"2024-09-21T23:27:15.272688+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:27:15.272688+0000","last_undegraded":"2024-09-21T23:27:15.272688+0000","last_fullsized":"2024-09-21T23:27:15.272688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'146","last_scrub_stamp":"2024-09-21T23:27:15.272612+0000","last_deep_scrub":"59'146","last_deep_scrub_stamp":"2024-09-21T23:27:15.272612+0000","last_clean_scrub_stamp":"2024-09-21T23:27:15.272612+0000","objects_scrubbed":1,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.239753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055139900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":636,"num_read_kb":592,"num_write":64,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"59'154","reported_seq":657,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197175+0000","last_change":"2024-09-21T23:26:47.963590+0000","last_active":"2024-09-21T23:27:08.197175+0000","last_peered":"2024-09-21T23:27:08.197175+0000","last_clean":"2024-09-21T23:27:08.197175+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:27:08.197175+0000","last_undegraded":"2024-09-21T23:27:08.197175+0000","last_fullsized":"2024-09-21T23:27:08.197175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:24:23.124579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063174900000000003,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":655,"num_read_kb":658,"num_write":67,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"59'124","reported_seq":463,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.027116+0000","last_change":"2024-09-21T23:26:47.912343+0000","last_active":"2024-09-21T23:27:08.027116+0000","last_peered":"2024-09-21T23:27:08.027116+0000","last_clean":"2024-09-21T23:27:08.027116+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:27:08.027116+0000","last_undegraded":"2024-09-21T23:27:08.027116+0000","last_fullsized":"2024-09-21T23:27:08.027116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046220700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":385,"num_read_kb":351,"num_write":58,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"59'80","reported_seq":242,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.424773+0000","last_change":"2024-09-21T23:26:48.000677+0000","last_active":"2024-09-21T23:27:08.424773+0000","last_peered":"2024-09-21T23:27:08.424773+0000","last_clean":"2024-09-21T23:27:08.424773+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:27:08.424773+0000","last_undegraded":"2024-09-21T23:27:08.424773+0000","last_fullsized":"2024-09-21T23:27:08.424773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:57:46.982872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000365359,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":22,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"59'119","reported_seq":324,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:08.197178+0000","last_change":"2024-09-21T23:26:47.963368+0000","last_active":"2024-09-21T23:27:08.197178+0000","last_peered":"2024-09-21T23:27:08.197178+0000","last_clean":"2024-09-21T23:27:08.197178+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:27:08.197178+0000","last_undegraded":"2024-09-21T23:27:08.197178+0000","last_fullsized":"2024-09-21T23:27:08.197178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:25:27.143894+0000","last_clean_scrub_stamp":"2024-09-21T23:25:27.143894+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:15:22.026082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036157399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":132,"num_write":67,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'119","reported_seq":238,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:14.953805+0000","last_change":"2024-09-21T23:27:14.953843+0000","last_active":"2024-09-21T23:27:14.953805+0000","last_peered":"2024-09-21T23:27:14.953805+0000","last_clean":"2024-09-21T23:27:14.953805+0000","last_became_active":"2024-09-21T23:26:54.861854+0000","last_became_peered":"2024-09-21T23:26:54.861854+0000","last_unstale":"2024-09-21T23:27:14.953805+0000","last_undegraded":"2024-09-21T23:27:14.953805+0000","last_fullsized":"2024-09-21T23:27:14.953805+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'119","last_scrub_stamp":"2024-09-21T23:27:14.953714+0000","last_deep_scrub":"76'119","last_deep_scrub_stamp":"2024-09-21T23:27:14.953714+0000","last_clean_scrub_stamp":"2024-09-21T23:27:14.953714+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:22:05.303156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2848,"num_read_kb":2705,"num_write":420,"num_write_kb":428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":978,"ondisk_log_size":978,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":47842,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542172,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32052,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339788,"statfs":{"total":96636764160,"available":96603942912,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509469,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":888832,"data_stored":375471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542171,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509468,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509469,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":929792,"data_stored":965839,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509468,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":925696,"data_stored":965820,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32052,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339788,"statfs":{"total":96636764160,"available":96603942912,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32052,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339788,"statfs":{"total":96636764160,"available":96603942912,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:27:18.229 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-21T23:25:27.143894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=25, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=27, tm_sec=13, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T23:27:18.230 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-21T23:25:27.143894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=25, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=27, tm_sec=13, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T23:27:18.230 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-21T23:25:27.143894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=25, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=27, tm_sec=13, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T23:27:18.230 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-21T23:25:27.143894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=25, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=27, tm_sec=13, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T23:27:18.230 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-21T23:25:27.143894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=25, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=27, tm_sec=13, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T23:27:18.230 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-21T23:25:27.143894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=25, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=27, tm_sec=13, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T23:27:18.230 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-21T23:25:27.143894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=25, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=27, tm_sec=13, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T23:27:18.230 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-21T23:27:38.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T23:27:38.532 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:27:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:27:38.545 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-21T23:27:37.799601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2964,"num_read_kb":2805,"num_write":579,"num_write_kb":2844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1099,"ondisk_log_size":1099,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256484,"kb_used_data":7000,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718236,"statfs":{"total":773094113280,"available":772831473664,"internally_reserved":0,"allocated":7168000,"data_stored":4184390,"data_compressed":19550,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002629"},"pg_stats":[{"pgid":"2.7","version":"59'114","reported_seq":430,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:17.749091+0000","last_change":"2024-09-21T23:27:17.749136+0000","last_active":"2024-09-21T23:27:17.749091+0000","last_peered":"2024-09-21T23:27:17.749091+0000","last_clean":"2024-09-21T23:27:17.749091+0000","last_became_active":"2024-09-21T23:25:28.165887+0000","last_became_peered":"2024-09-21T23:25:28.165887+0000","last_unstale":"2024-09-21T23:27:17.749091+0000","last_undegraded":"2024-09-21T23:27:17.749091+0000","last_fullsized":"2024-09-21T23:27:17.749091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'114","last_scrub_stamp":"2024-09-21T23:27:17.748993+0000","last_deep_scrub":"59'114","last_deep_scrub_stamp":"2024-09-21T23:27:17.748993+0000","last_clean_scrub_stamp":"2024-09-21T23:27:17.748993+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:51:29.972690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002039819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":357,"num_read_kb":326,"num_write":43,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"59'86","reported_seq":282,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:17.124354+0000","last_change":"2024-09-21T23:27:17.124378+0000","last_active":"2024-09-21T23:27:17.124354+0000","last_peered":"2024-09-21T23:27:17.124354+0000","last_clean":"2024-09-21T23:27:17.124354+0000","last_became_active":"2024-09-21T23:26:17.546387+0000","last_became_peered":"2024-09-21T23:26:17.546387+0000","last_unstale":"2024-09-21T23:27:17.124354+0000","last_undegraded":"2024-09-21T23:27:17.124354+0000","last_fullsized":"2024-09-21T23:27:17.124354+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'86","last_scrub_stamp":"2024-09-21T23:27:17.124273+0000","last_deep_scrub":"59'86","last_deep_scrub_stamp":"2024-09-21T23:27:17.124273+0000","last_clean_scrub_stamp":"2024-09-21T23:27:17.124273+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:40:08.993327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000423046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":218,"num_read_kb":168,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"59'155","reported_seq":520,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:19.447918+0000","last_change":"2024-09-21T23:27:19.447918+0000","last_active":"2024-09-21T23:27:19.447918+0000","last_peered":"2024-09-21T23:27:19.447918+0000","last_clean":"2024-09-21T23:27:19.447918+0000","last_became_active":"2024-09-21T23:25:28.441879+0000","last_became_peered":"2024-09-21T23:25:28.441879+0000","last_unstale":"2024-09-21T23:27:19.447918+0000","last_undegraded":"2024-09-21T23:27:19.447918+0000","last_fullsized":"2024-09-21T23:27:19.447918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'155","last_scrub_stamp":"2024-09-21T23:27:19.447788+0000","last_deep_scrub":"59'155","last_deep_scrub_stamp":"2024-09-21T23:27:19.447788+0000","last_clean_scrub_stamp":"2024-09-21T23:27:19.447788+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:51:52.635192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018417240000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":432,"num_read_kb":441,"num_write":88,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"59'146","reported_seq":551,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:15.272711+0000","last_change":"2024-09-21T23:27:15.272711+0000","last_active":"2024-09-21T23:27:15.272711+0000","last_peered":"2024-09-21T23:27:15.272711+0000","last_clean":"2024-09-21T23:27:15.272711+0000","last_became_active":"2024-09-21T23:25:28.441769+0000","last_became_peered":"2024-09-21T23:25:28.441769+0000","last_unstale":"2024-09-21T23:27:15.272711+0000","last_undegraded":"2024-09-21T23:27:15.272711+0000","last_fullsized":"2024-09-21T23:27:15.272711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'146","last_scrub_stamp":"2024-09-21T23:27:15.272612+0000","last_deep_scrub":"59'146","last_deep_scrub_stamp":"2024-09-21T23:27:15.272612+0000","last_clean_scrub_stamp":"2024-09-21T23:27:15.272612+0000","objects_scrubbed":1,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:35:46.239753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055139900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":636,"num_read_kb":592,"num_write":64,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"76'156","reported_seq":669,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:18.543444+0000","last_change":"2024-09-21T23:27:18.543444+0000","last_active":"2024-09-21T23:27:18.543444+0000","last_peered":"2024-09-21T23:27:18.543444+0000","last_clean":"2024-09-21T23:27:18.543444+0000","last_became_active":"2024-09-21T23:25:28.167164+0000","last_became_peered":"2024-09-21T23:25:28.167164+0000","last_unstale":"2024-09-21T23:27:18.543444+0000","last_undegraded":"2024-09-21T23:27:18.543444+0000","last_fullsized":"2024-09-21T23:27:18.543444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'156","last_scrub_stamp":"2024-09-21T23:27:18.543307+0000","last_deep_scrub":"76'156","last_deep_scrub_stamp":"2024-09-21T23:27:18.543307+0000","last_clean_scrub_stamp":"2024-09-21T23:27:18.543307+0000","objects_scrubbed":2,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:02:40.904930+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00063174900000000003,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":655,"num_read_kb":658,"num_write":67,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"59'124","reported_seq":472,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:16.109347+0000","last_change":"2024-09-21T23:27:16.109347+0000","last_active":"2024-09-21T23:27:16.109347+0000","last_peered":"2024-09-21T23:27:16.109347+0000","last_clean":"2024-09-21T23:27:16.109347+0000","last_became_active":"2024-09-21T23:25:28.166257+0000","last_became_peered":"2024-09-21T23:25:28.166257+0000","last_unstale":"2024-09-21T23:27:16.109347+0000","last_undegraded":"2024-09-21T23:27:16.109347+0000","last_fullsized":"2024-09-21T23:27:16.109347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'124","last_scrub_stamp":"2024-09-21T23:27:16.109202+0000","last_deep_scrub":"59'124","last_deep_scrub_stamp":"2024-09-21T23:27:16.109202+0000","last_clean_scrub_stamp":"2024-09-21T23:27:16.109202+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:28:37.285053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046220700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":385,"num_read_kb":351,"num_write":58,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"59'80","reported_seq":252,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:18.484575+0000","last_change":"2024-09-21T23:27:18.484575+0000","last_active":"2024-09-21T23:27:18.484575+0000","last_peered":"2024-09-21T23:27:18.484575+0000","last_clean":"2024-09-21T23:27:18.484575+0000","last_became_active":"2024-09-21T23:25:28.165500+0000","last_became_peered":"2024-09-21T23:25:28.165500+0000","last_unstale":"2024-09-21T23:27:18.484575+0000","last_undegraded":"2024-09-21T23:27:18.484575+0000","last_fullsized":"2024-09-21T23:27:18.484575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'80","last_scrub_stamp":"2024-09-21T23:27:18.484435+0000","last_deep_scrub":"59'80","last_deep_scrub_stamp":"2024-09-21T23:27:18.484435+0000","last_clean_scrub_stamp":"2024-09-21T23:27:18.484435+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:31:50.925529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000365359,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":22,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"59'119","reported_seq":334,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:17.515217+0000","last_change":"2024-09-21T23:27:17.515217+0000","last_active":"2024-09-21T23:27:17.515217+0000","last_peered":"2024-09-21T23:27:17.515217+0000","last_clean":"2024-09-21T23:27:17.515217+0000","last_became_active":"2024-09-21T23:25:28.166600+0000","last_became_peered":"2024-09-21T23:25:28.166600+0000","last_unstale":"2024-09-21T23:27:17.515217+0000","last_undegraded":"2024-09-21T23:27:17.515217+0000","last_fullsized":"2024-09-21T23:27:17.515217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'119","last_scrub_stamp":"2024-09-21T23:27:17.515082+0000","last_deep_scrub":"59'119","last_deep_scrub_stamp":"2024-09-21T23:27:17.515082+0000","last_clean_scrub_stamp":"2024-09-21T23:27:17.515082+0000","objects_scrubbed":1,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:49:30.780194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036157399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":132,"num_write":67,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'119","reported_seq":238,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T23:27:14.953805+0000","last_change":"2024-09-21T23:27:14.953843+0000","last_active":"2024-09-21T23:27:14.953805+0000","last_peered":"2024-09-21T23:27:14.953805+0000","last_clean":"2024-09-21T23:27:14.953805+0000","last_became_active":"2024-09-21T23:26:54.861854+0000","last_became_peered":"2024-09-21T23:26:54.861854+0000","last_unstale":"2024-09-21T23:27:14.953805+0000","last_undegraded":"2024-09-21T23:27:14.953805+0000","last_fullsized":"2024-09-21T23:27:14.953805+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'119","last_scrub_stamp":"2024-09-21T23:27:14.953714+0000","last_deep_scrub":"76'119","last_deep_scrub_stamp":"2024-09-21T23:27:14.953714+0000","last_clean_scrub_stamp":"2024-09-21T23:27:14.953714+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:22:05.303156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2848,"num_read_kb":2705,"num_write":420,"num_write_kb":428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":980,"ondisk_log_size":980,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":47842,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542176,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32048,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509473,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32052,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339788,"statfs":{"total":96636764160,"available":96603942912,"internally_reserved":0,"allocated":888832,"data_stored":375471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542175,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509472,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32048,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509473,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":929792,"data_stored":965839,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509472,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":925696,"data_stored":965820,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32048,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32052,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339788,"statfs":{"total":96636764160,"available":96603942912,"internally_reserved":0,"allocated":884736,"data_stored":375452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":9775,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":8742,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:27:38.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-21T23:27:38.852 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-21T23:27:38.852 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-21T23:27:38.852 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-21T23:27:38.852 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:38.898 INFO:tasks.ceph.osd.0:Stopped 2024-09-21T23:27:38.899 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-21T23:27:38.899 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:38.934 INFO:tasks.ceph.osd.1:Stopped 2024-09-21T23:27:38.934 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-21T23:27:38.934 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:39.001 INFO:tasks.ceph.osd.2:Stopped 2024-09-21T23:27:39.001 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-21T23:27:39.001 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:39.077 INFO:tasks.ceph.osd.3:Stopped 2024-09-21T23:27:39.077 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-21T23:27:39.078 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:39.115 INFO:tasks.ceph.osd.4:Stopped 2024-09-21T23:27:39.116 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-21T23:27:39.116 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:39.193 INFO:tasks.ceph.osd.5:Stopped 2024-09-21T23:27:39.193 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-21T23:27:39.193 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:39.260 INFO:tasks.ceph.osd.6:Stopped 2024-09-21T23:27:39.260 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-21T23:27:39.260 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:39.325 INFO:tasks.ceph.osd.7:Stopped 2024-09-21T23:27:39.325 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-21T23:27:39.326 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-21T23:27:39.326 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:39.350 INFO:tasks.ceph.mgr.y:Stopped 2024-09-21T23:27:39.350 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-21T23:27:39.350 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:39.395 INFO:tasks.ceph.mgr.x:Stopped 2024-09-21T23:27:39.395 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-21T23:27:39.395 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-21T23:27:39.395 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:39.406 INFO:tasks.ceph.mon.a:Stopped 2024-09-21T23:27:39.406 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-21T23:27:39.406 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:39.457 INFO:tasks.ceph.mon.c:Stopped 2024-09-21T23:27:39.457 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-21T23:27:39.457 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T23:27:39.469 INFO:tasks.ceph.mon.b:Stopped 2024-09-21T23:27:39.470 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-21T23:27:39.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-21T23:27:39.534 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:27:39.534 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-21T23:27:39.747 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:27:39.747 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-21T23:27:39.822 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:27:39.822 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-21T23:27:39.878 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:27:39.879 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-21T23:27:39.929 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:27:39.930 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-21T23:27:40.065 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:27:40.065 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-21T23:27:40.136 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:27:40.136 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-21T23:27:40.187 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:27:40.187 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-21T23:27:40.244 INFO:tasks.ceph:Archiving mon data... 2024-09-21T23:27:40.245 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523/data/mon.a.tgz 2024-09-21T23:27:40.245 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-21T23:27:40.261 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.94HIEGF7py 2024-09-21T23:27:40.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.94HIEGF7py 2024-09-21T23:27:40.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.94HIEGF7py 2024-09-21T23:27:40.475 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.94HIEGF7py is 286KB 2024-09-21T23:27:40.570 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.94HIEGF7py 2024-09-21T23:27:40.585 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523/data/mon.c.tgz 2024-09-21T23:27:40.585 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-21T23:27:40.641 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.mVVWhjiK66 2024-09-21T23:27:40.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.mVVWhjiK66 2024-09-21T23:27:40.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.mVVWhjiK66 2024-09-21T23:27:40.864 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.mVVWhjiK66 is 395KB 2024-09-21T23:27:40.959 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.mVVWhjiK66 2024-09-21T23:27:40.975 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523/data/mon.b.tgz 2024-09-21T23:27:40.975 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-09-21T23:27:40.994 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.sZj0q7IxGS 2024-09-21T23:27:40.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.sZj0q7IxGS 2024-09-21T23:27:41.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.sZj0q7IxGS 2024-09-21T23:27:41.245 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.sZj0q7IxGS is 395KB 2024-09-21T23:27:41.313 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.sZj0q7IxGS 2024-09-21T23:27:41.329 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-21T23:27:41.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T23:27:41.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T23:27:41.411 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T23:27:41.451 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T23:27:41.452 INFO:tasks.ceph:Archiving crash dumps... 2024-09-21T23:27:41.454 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523/remote/smithi031/crash 2024-09-21T23:27:41.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T23:27:41.480 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523/remote/smithi060/crash 2024-09-21T23:27:41.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T23:27:41.517 INFO:tasks.ceph:Compressing logs... 2024-09-21T23:27:41.518 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T23:27:41.525 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T23:27:41.547 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-21T23:27:41.547 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51876.log 2024-09-21T23:27:41.548 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52128.log 2024-09-21T23:27:41.549 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.51876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37880.log 2024-09-21T23:27:41.549 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51133.log 2024-09-21T23:27:41.549 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38298.log 2024-09-21T23:27:41.549 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51409.log 2024-09-21T23:27:41.589 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51876.log.gz 2024-09-21T23:27:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52128.log: /var/log/ceph/ceph-client.admin.37880.log: /var/log/ceph/ceph-client.admin.38298.log: /var/log/ceph/ceph-client.admin.51133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46856.log 2024-09-21T23:27:41.589 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37880.log.gz 2024-09-21T23:27:41.589 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38298.log.gz 2024-09-21T23:27:41.589 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51133.log.gz 2024-09-21T23:27:41.589 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52128.log.gz 2024-09-21T23:27:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40758.log 2024-09-21T23:27:41.589 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51409.log.gz 2024-09-21T23:27:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52714.log 2024-09-21T23:27:41.589 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46856.log.gz 2024-09-21T23:27:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40183.log 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40758.log.gz 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38578.log 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52714.log.gz 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37318.log 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37402.log 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38578.log.gz 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39822.log 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37318.log.gz 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48194.log 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37402.log.gz 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51074.log 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39822.log.gz 2024-09-21T23:27:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48194.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.48194.log.gz -5 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38902.log 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37813.log 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51074.log.gz 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48122.log 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38902.log.gz 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51918.log 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37813.log.gz 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38135.log 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48122.log.gz 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40142.log 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51918.log.gz 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48040.log 2024-09-21T23:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38135.log.gz 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52590.log 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40142.log.gz 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48338.log 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr: 51.3% -- replaced with /var/log/ceph/ceph-client.admin.48040.log.gz 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38136.log 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52590.log.gz 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50490.log 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48338.log.gz 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38136.log.gz 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50792.log 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37360.log 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50490.log.gz 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48094.log 2024-09-21T23:27:41.592 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50792.log.gz 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49267.log 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37360.log.gz 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39823.log 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr: 52.9% -- replaced with /var/log/ceph/ceph-client.admin.48094.log.gz 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50154.log 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49267.log.gz 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39823.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39823.log.gz 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52044.log 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50154.log.gz 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41242.log 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43606.log 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52044.log.gz 2024-09-21T23:27:41.593 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41242.log.gz 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33285.log 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44870.log 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43606.log.gz 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.33285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48246.log 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33285.log.gz 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38704.log 0.0% 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44870.log.gz 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49008.log 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48246.log.gz 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47900.log 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38704.log.gz 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38452.log 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49008.log.gz 2024-09-21T23:27:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39603.log 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.47900.log.gz 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47648.log 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38452.log.gz 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46335.log 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39603.log.gz 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52170.log 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47648.log.gz 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46335.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.46335.log.gz --verbose 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.37569.log 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50070.log 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52170.log.gz 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37569.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.37569.log.gz -- 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.51960.log 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37267.log 2024-09-21T23:27:41.595 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50070.log.gz 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51072.log 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51960.log.gz 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37267.log.gz 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44672.log 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47921.log 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51072.log.gz 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49304.log 0.0% 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44672.log.gz 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49324.log 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.47921.log.gz 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38536.log 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49304.log.gz 2024-09-21T23:27:41.596 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49860.log 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49324.log.gz 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37931.log 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38536.log.gz 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51708.log 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49860.log.gz 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46209.log 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37931.log.gz 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51708.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.51708.log.gz /var/log/ceph/ceph-client.admin.48929.log 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45554.log 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46209.log.gz 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43004.log 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48929.log.gz 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51750.log 2024-09-21T23:27:41.597 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45554.log.gz 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51410.log 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43004.log.gz 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51750.log.gz 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49963.log 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52632.log 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51410.log.gz 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49963.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49963.log.gz -5 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47608.log 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38766.log 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52632.log.gz 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47608.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27449.log 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.47608.log.gz 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39365.log 2024-09-21T23:27:41.598 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38766.log.gz 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.tmp-client.admin.27449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48450.log 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27449.log.gz 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49494.log 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39365.log.gz 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46512.log 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48450.log.gz 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50280.log 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr: 54.4% -- replaced with /var/log/ceph/ceph-client.admin.49494.log.gz 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52506.log 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46512.log.gz 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47210.log 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50280.log.gz 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52506.log.gz 2024-09-21T23:27:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51408.log 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38620.log 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47210.log.gz 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47588.log 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51408.log.gz 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41180.log 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38620.log.gz 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47588.log: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.47588.log.gz 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47783.log 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48720.log 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41180.log.gz 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47168.log 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47783.log.gz 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50798.log 2024-09-21T23:27:41.600 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48720.log.gz 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47168.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47168.log.gz 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44508.log 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50798.log.gz 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44445.log 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51134.log 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44508.log.gz 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44425.log 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44445.log.gz 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46857.log 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51134.log.gz 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47400.log 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44425.log.gz 2024-09-21T23:27:41.601 INFO:teuthology.orchestra.run.smithi031.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47878.log 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46857.log.gz 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43646.log 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47400.log.gz 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47961.log 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47878.log.gz 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43646.log.gz 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52254.log 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40400.log 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47961.log.gz 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52254.log.gz 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44932.log 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42387.log 2024-09-21T23:27:41.602 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40400.log.gz 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42221.log 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44932.log.gz 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45452.log 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42387.log.gz 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49049.log 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42221.log.gz 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47314.log 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45452.log.gz 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38662.log 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49049.log.gz 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.47314.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49635.log 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47314.log.gz 2024-09-21T23:27:41.603 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38662.log.gz 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37097.log 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5/var/log/ceph/ceph-client.admin.49635.log: --verbose -- /var/log/ceph/ceph-client.admin.50364.log 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49635.log.gz 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37097.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37097.log.gz -5 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47229.log 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.50364.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41359.log 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50364.log.gz 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43789.log 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47229.log.gz 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42042.log 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41359.log.gz 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51834.log 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43789.log.gz 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50797.log 2024-09-21T23:27:41.604 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42042.log.gz 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48813.log 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51834.log.gz 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45231.log 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50797.log.gz 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47804.log 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48813.log.gz 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41880.log/var/log/ceph/ceph-client.admin.45231.log: 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45231.log.gz 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37581.log 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.47804.log.gz 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52212.log 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41880.log.gz 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37581.log.gz 2024-09-21T23:27:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43285.log 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49091.log 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52212.log.gz 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43285.log.gz 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38219.log 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47378.log 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49091.log.gz 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38219.log.gz 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37242.log 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50532.log 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47378.log.gz 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37242.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37242.log.gz -5 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52086.log 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47097.log 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50532.log.gz 2024-09-21T23:27:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42887.log 2024-09-21T23:27:41.607 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52086.log.gz 2024-09-21T23:27:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-21T23:27:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47097.log.gz 2024-09-21T23:27:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37574.log 2024-09-21T23:27:41.607 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42887.log.gz 2024-09-21T23:27:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46272.log 2024-09-21T23:27:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37584.log 2024-09-21T23:27:41.607 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37574.log.gz 2024-09-21T23:27:41.615 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33081.log 2024-09-21T23:27:41.615 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33241.log 2024-09-21T23:27:41.615 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33081.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-21T23:27:41.615 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33081.log.gz 2024-09-21T23:27:41.615 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33241.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-21T23:27:41.615 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33241.log.gz 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33321.log 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33281.log 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33321.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.33321.log.gz --verbose 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/ceph-osd.6.log 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33281.log.gz 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.6.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33121.log 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33365.log 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33201.log 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33121.log.gz 2024-09-21T23:27:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33365.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-21T23:27:41.617 INFO:teuthology.orchestra.run.smithi060.stderr: 85.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-21T23:27:41.617 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33365.log.gz 2024-09-21T23:27:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33201.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-21T23:27:41.617 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33201.log.gz 2024-09-21T23:27:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33161.log 2024-09-21T23:27:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.33161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33161.log.gz 2024-09-21T23:27:41.638 INFO:teuthology.orchestra.run.smithi060.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-21T23:27:41.650 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-21T23:27:41.675 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45412.log 2024-09-21T23:27:41.675 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46272.log.gz 2024-09-21T23:27:41.675 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37584.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.39098.log -- replaced with /var/log/ceph/ceph-client.admin.37584.log.gz 2024-09-21T23:27:41.675 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-21T23:27:41.675 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52464.log 2024-09-21T23:27:41.675 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45412.log.gz 2024-09-21T23:27:41.675 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39098.log.gz 2024-09-21T23:27:41.675 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48277.log 2024-09-21T23:27:41.675 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40962.log 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52464.log.gz 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52548.log 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr: 52.8% -- replaced with /var/log/ceph/ceph-client.admin.48277.log.gz 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.40962.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37186.log 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40962.log.gz 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52548.log.gz 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50574.log 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50238.log 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37186.log.gz 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39202.log 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50574.log.gz 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41484.log 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50238.log.gz 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39202.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.47689.log -- replaced with /var/log/ceph/ceph-client.admin.39202.log.gz 2024-09-21T23:27:41.676 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39761.log 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41484.log.gz 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47689.log.gz 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40580.log 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52338.log 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.39761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39761.log.gz 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42684.log 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40580.log.gz 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.52338.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52674.log 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52338.log.gz 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43487.log 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42684.log.gz 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44368.log 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52674.log.gz 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47207.log 2024-09-21T23:27:41.677 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43487.log.gz 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47277.log 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44368.log.gz 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48396.log 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47207.log.gz 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48591.log 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47277.log.gz 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37573.log/var/log/ceph/ceph-client.admin.48396.log: 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.48396.log.gz 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42343.log 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48591.log.gz 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47556.log 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37573.log.gz 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44049.log 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42343.log.gz 2024-09-21T23:27:41.678 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47556.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47556.log.gz 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44049.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.38494.log -- replaced with /var/log/ceph/ceph-client.admin.44049.log.gz 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50786.log 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.38494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44029.log 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38494.log.gz 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45328.log 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50786.log.gz 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49880.log 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44029.log.gz 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48908.log 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45328.log.gz 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51132.log 2024-09-21T23:27:41.679 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49880.log.gz 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37818.log 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr: 54.8% -- replaced with /var/log/ceph/ceph-client.admin.48908.log.gz 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.51132.log: -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51132.log.gz 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.41524.log 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37818.log.gz 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52296.log 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37582.log 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41524.log.gz 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52296.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52296.log.gz 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38156.log 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37582.log.gz 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50795.log 2024-09-21T23:27:41.680 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51468.log 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38156.log: gzip/var/log/ceph/ceph-client.admin.50795.log: -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.43001.log 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50795.log.gz 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38156.log.gz 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51468.log.gz 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50112.log 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47742.log 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43001.log.gz 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50112.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50112.log.gz 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49126.log 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47742.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42117.log 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47742.log.gz 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49126.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46208.log 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49126.log.gz 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49195.log 2024-09-21T23:27:41.681 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42117.log: /var/log/ceph/ceph-client.admin.46208.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42117.log.gz 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46208.log.gz 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37874.log 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49399.log 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49195.log.gz 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37874.log.gz 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48991.log 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49226.log 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48991.log: /var/log/ceph/ceph-client.admin.49399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49399.log.gz 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48991.log.gz 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39422.log 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-21T23:27:41.682 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49226.log: 92.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr: 53.8%/var/log/ceph/ceph-client.admin.39422.log: -- replaced with /var/log/ceph/ceph-client.admin.49226.log.gz 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39422.log.gz 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45370.log 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51406.log 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47668.log 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45370.log.gz 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51406.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47710.log 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51406.log.gz 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr: 91.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48970.log 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47668.log: 0.0%/var/log/ceph/ceph-client.admin.47710.log: -- replaced with /var/log/ceph/ceph-client.admin.47668.log.gz 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45069.log 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr: 6.4% -- replaced with /var/log/ceph/ceph-client.admin.47710.log.gz 2024-09-21T23:27:41.683 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49028.log 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr: 53.0% -- replaced with /var/log/ceph/ceph-client.admin.48970.log.gz 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.45069.log: -- /var/log/ceph/ceph-client.admin.50322.log 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45069.log.gz 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49028.log: 53.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38217.log 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49028.log.gz 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41038.log 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50322.log.gz 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47515.log 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38217.log: /var/log/ceph/ceph-client.admin.41038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50196.log 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38217.log.gz 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41038.log.gz 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47515.log.gz 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51470.log 2024-09-21T23:27:41.684 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49346.log 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50196.log.gz 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37820.log 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51470.log: /var/log/ceph/ceph-client.admin.49346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51470.log.gz 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49346.log.gz 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38901.log 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37820.log.gz 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48650.log 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38901.log.gz 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49818.log 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48650.log.gz 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45793.log 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49818.log.gz 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50448.log 2024-09-21T23:27:41.685 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45793.log.gz 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50616.log 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50448.log.gz 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50789.log 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50616.log.gz 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44550.log 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50789.log.gz 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40622.log 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44550.log.gz 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38294.log 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40622.log.gz 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48716.log 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38294.log.gz 2024-09-21T23:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37572.log 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48716.log.gz 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33683.log 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.37572.log: -- /var/log/ceph/ceph-client.admin.48162.log 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37572.log.gz 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.33683.log: /var/log/ceph/ceph-client.admin.51471.log 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33683.log.gz 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48162.log.gz 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48700.log 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47043.log 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51471.log.gz 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48845.log 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48700.log: /var/log/ceph/ceph-client.admin.47043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48700.log.gz 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47043.log.gz 2024-09-21T23:27:41.687 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43729.log 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48845.log.gz 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50796.log 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43729.log.gz 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47484.log 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50796.log.gz 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52380.log 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47484.log.gz 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40436.log 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52380.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47779.log 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52380.log.gz 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.40436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37933.log 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40436.log.gz 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47337.log 2024-09-21T23:27:41.688 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47779.log.gz 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.37933.log: -- /var/log/ceph/ceph-client.admin.47020.log 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37933.log.gz 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47337.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47337.log.gz 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37583.log 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50787.log 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47020.log: /var/log/ceph/ceph-client.admin.37583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42303.log 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37583.log.gz 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47020.log.gz 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50787.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50787.log.gz 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51071.log 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42303.log.gz 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49771.log 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51071.log.gz 2024-09-21T23:27:41.689 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51330.log 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49771.log.gz 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38905.log 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.51330.log: -- /var/log/ceph/ceph-client.admin.47857.log 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51330.log.gz 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.38905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38905.log.gz 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51666.log 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47857.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.43168.log 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47857.log.gz 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51666.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51666.log.gz 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47128.log 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43168.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38222.log 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43168.log.gz 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48418.log 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47128.log.gz 2024-09-21T23:27:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38222.log: -- /var/log/ceph/ceph-client.admin.51792.log 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38222.log.gz 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48418.log.gz 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33576.log 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46984.log 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51792.log: /var/log/ceph/ceph-client.admin.33576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42728.log 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51792.log.gz 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33576.log.gz 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41442.log 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46984.log.gz 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42728.log: -- /var/log/ceph/ceph-client.admin.43564.log 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42728.log.gz/var/log/ceph/ceph-client.admin.41442.log: 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47080.log 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2024-09-21T23:27:41.691 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45834.log 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43564.log: 0.0%/var/log/ceph/ceph-client.admin.47080.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47080.log.gz 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.48502.log 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43564.log.gz 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45834.log.gz 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51073.log 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48502.log.gz 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51058.log 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51073.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.50406.log 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51073.log.gz 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46655.log 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51058.log.gz 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50406.log: -- /var/log/ceph/ceph-client.admin.48574.log 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50406.log.gz 2024-09-21T23:27:41.692 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.46655.log: -- /var/log/ceph/ceph-client.admin.49457.log 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46655.log.gz 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48574.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48949.log 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48574.log.gz 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49457.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49457.log.gz 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37879.log 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48949.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52422.log 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48949.log.gz 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.37879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37879.log.gz 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48533.log 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52422.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52002.log 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52422.log.gz 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48355.log 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48533.log: /var/log/ceph/ceph-client.admin.52002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52002.log.gz 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr: 53.2% -- replaced with /var/log/ceph/ceph-client.admin.48533.log.gz 2024-09-21T23:27:41.693 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48877.log 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48744.log 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48877.log: /var/log/ceph/ceph-client.admin.48355.log: 0.0% 24.6% -- replaced with /var/log/ceph/ceph-client.admin.48877.log.gz -- replaced with /var/log/ceph/ceph-client.admin.48355.log.gz 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49964.log 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47422.log 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48744.log: /var/log/ceph/ceph-client.admin.49964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48744.log.gz 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49964.log.gz 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50028.log 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47422.log.gz 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46072.log 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41823.log 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46072.log: /var/log/ceph/ceph-client.admin.50028.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50028.log.gz 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47628.log 2024-09-21T23:27:41.694 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46072.log.gz 2024-09-21T23:27:41.695 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41823.log.gz 2024-09-21T23:27:41.695 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47628.log.gz 2024-09-21T23:27:41.700 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-21T23:27:41.702 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-21T23:27:41.706 INFO:teuthology.orchestra.run.smithi031.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-21T23:27:41.714 INFO:teuthology.orchestra.run.smithi031.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-21T23:27:41.728 INFO:teuthology.orchestra.run.smithi031.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-21T23:27:41.758 INFO:teuthology.orchestra.run.smithi060.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-21T23:27:41.758 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-21T23:27:41.760 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-21T23:27:41.760 INFO:teuthology.orchestra.run.smithi060.stderr:real 0m0.184s 2024-09-21T23:27:41.760 INFO:teuthology.orchestra.run.smithi060.stderr:user 0m0.663s 2024-09-21T23:27:41.760 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m0.041s 2024-09-21T23:27:41.783 INFO:teuthology.orchestra.run.smithi031.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-21T23:27:41.795 INFO:teuthology.orchestra.run.smithi031.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-21T23:27:41.797 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-21T23:27:41.797 INFO:teuthology.orchestra.run.smithi031.stderr:real 0m0.260s 2024-09-21T23:27:41.797 INFO:teuthology.orchestra.run.smithi031.stderr:user 0m0.909s 2024-09-21T23:27:41.797 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m0.179s 2024-09-21T23:27:41.798 INFO:tasks.ceph:Archiving logs... 2024-09-21T23:27:41.798 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523/remote/smithi031/log 2024-09-21T23:27:41.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T23:27:42.152 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523/remote/smithi060/log 2024-09-21T23:27:42.153 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T23:27:42.275 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T23:27:42.298 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T23:27:42.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T23:27:42.301 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T23:27:42.376 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T23:27:42.397 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T23:27:42.397 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T23:27:42.400 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T23:27:42.413 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-09-21T23:27:42.416 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-09-21T23:27:42.419 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T23:27:42.420 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-09-21T23:27:42.420 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv01.front.sepia.ceph.com 3 6 377 60 +378us[ +371us] +/- 47ms 2024-09-21T23:27:42.420 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 -2489us[-2496us] +/- 65ms 2024-09-21T23:27:42.420 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +1885us[+1878us] +/- 62ms 2024-09-21T23:27:42.420 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T23:27:42.421 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T23:27:42.421 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-09-21T23:27:42.422 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 +393us[ +396us] +/- 47ms 2024-09-21T23:27:42.422 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 -2497us[-2497us] +/- 65ms 2024-09-21T23:27:42.422 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +1897us[+1897us] +/- 62ms 2024-09-21T23:27:42.422 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T23:27:42.424 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T23:27:42.481 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T23:27:42.481 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T23:27:42.509 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T23:27:42.584 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T23:27:42.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T23:27:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:27:42.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T23:27:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:27:42.714 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T23:27:42.723 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T23:27:42.759 INFO:teuthology.task.internal:Duration was 554.387056 seconds 2024-09-21T23:27:42.759 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T23:27:42.792 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T23:27:42.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T23:27:42.795 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T23:27:42.833 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T23:27:42.839 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T23:27:43.118 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T23:27:43.118 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:27:43.119 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T23:27:43.180 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:27:43.180 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T23:27:43.206 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T23:27:43.206 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T23:27:43.223 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T23:27:43.274 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T23:27:43.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T23:27:43.291 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T23:27:43.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T23:27:43.450 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T23:27:43.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T23:27:43.476 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T23:27:43.502 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T23:27:43.526 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T23:27:43.528 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T23:27:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-09-21T23:27:43.566 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-09-21T23:27:43.617 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T23:27:43.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:27:43.632 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T23:27:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:27:43.672 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T23:27:43.682 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T23:27:43.683 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523/remote/smithi031 2024-09-21T23:27:43.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T23:27:43.717 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914523/remote/smithi060 2024-09-21T23:27:43.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T23:27:43.751 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T23:27:43.752 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T23:27:43.755 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T23:27:43.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T23:27:43.812 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T23:27:43.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T23:27:43.834 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T23:27:43.835 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T23:27:43.837 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T23:27:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 23:27 /home/ubuntu/cephtest 2024-09-21T23:27:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: 265816 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 23:27 /home/ubuntu/cephtest 2024-09-21T23:27:43.858 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T23:27:43.867 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T23:27:43.937 INFO:teuthology.run:Summary data: description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/journal} duration: 554.3870558738708 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-21T23:27:43.937 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T23:27:44.017 INFO:teuthology.run:pass